2024-04-25T19:38:56.392 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-25T19:38:56.392 DEBUG:teuthology.run:Teuthology command: teuthology --description orch:cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_mgr_res_dom} --owner scheduled_phlogistonjohn@teuthology --archive /home/teuthworker/archive/phlogistonjohn-2024-04-25_11:39:08-orch:cephadm-wip-phlogistonjohn-testing-2024-04-24-1356-distro-default-smithi/7673419 --name phlogistonjohn-2024-04-25_11:39:08-orch:cephadm-wip-phlogistonjohn-testing-2024-04-24-1356-distro-default-smithi --verbose -- /home/teuthworker/archive/phlogistonjohn-2024-04-25_11:39:08-orch:cephadm-wip-phlogistonjohn-testing-2024-04-24-1356-distro-default-smithi/7673419/orig.config.yaml 2024-04-25T19:38:56.424 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T19:38:56.673 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/phlogistonjohn-2024-04-25_11:39:08-orch:cephadm-wip-phlogistonjohn-testing-2024-04-24-1356-distro-default-smithi/7673419 branch: wip-phlogistonjohn-testing-2024-04-24-1356 description: orch:cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_mgr_res_dom} email: jmulligan@redhat.com first_in_suite: false job_id: '7673419' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: phlogistonjohn-2024-04-25_11:39:08-orch:cephadm-wip-phlogistonjohn-testing-2024-04-24-1356-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-phlogistonjohn-testing-2024-04-24-1356 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-only-match: - CEPHADM_ sha1: b48599b46c4ce29785962a8af5ddf30f6f38e57f ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: b48599b46c4ce29785962a8af5ddf30f6f38e57f workunit: branch: wip-phlogistonjohn-testing-2024-04-24-1356 sha1: b48599b46c4ce29785962a8af5ddf30f6f38e57f owner: scheduled_phlogistonjohn@teuthology priority: 99 repo: https://git.ceph.com/ceph-ci.git roles: - - host.a - mon.a - mgr.x - osd.0 - osd.1 - client.0 - - host.b - cephadm.exclude seed: 9641 sha1: b48599b46c4ce29785962a8af5ddf30f6f38e57f sleep_before_teardown: 0 suite: orch:cephadm suite_branch: wip-phlogistonjohn-testing-2024-04-24-1356 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_b48599b46c4ce29785962a8af5ddf30f6f38e57f/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: b48599b46c4ce29785962a8af5ddf30f6f38e57f targets: smithi132.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGj9DBZof78QeX9azISYEVgx344AwuMrtGc0Fgm+apJWdf45t4JMXmdAhhvUlDlH3BvqnqiRyNJR+0QXAxjW15w= smithi150.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOJKSXfi3X07tQKGC42atfFsxAAdAfPbrIyzwQX4GLc6HAKdWy5vbDCvhkVmn7vI9cOf/PKSATmgJ4kwgn7PZ88= tasks: - cephadm.deploy_samba_ad_dc: role: host.b - cephadm: null - cephadm.shell: host.a: - ceph fs volume create cephfs - cephadm.wait_for_service: service: mds.cephfs - cephadm.shell: host.a: - cmd: ceph fs subvolumegroup create cephfs smb - cmd: ceph fs subvolume create cephfs sv1 --group-name=smb --mode=0777 - cmd: ceph fs subvolume create cephfs sv2 --group-name=smb --mode=0777 - cmd: ceph mgr module enable smb - cmd: sleep 30 - cmd: ceph smb apply -i - stdin: "# --- Begin Embedded YAML\n- resource_type: ceph.smb.cluster\n cluster_id:\ \ modtest1\n auth_mode: active-directory\n domain_settings:\n realm:\ \ DOMAIN1.SINK.TEST\n join_sources:\n - source_type: resource\n \ \ ref: join1-admin\n custom_dns:\n - \"192.168.76.204\"\n placement:\n\ \ count: 1\n- resource_type: ceph.smb.join.auth\n auth_id: join1-admin\n\ \ auth:\n username: Administrator\n password: Passw0rd\n- resource_type:\ \ ceph.smb.share\n cluster_id: modtest1\n share_id: share1\n cephfs:\n\ \ volume: cephfs\n subvolumegroup: smb\n subvolume: sv1\n path:\ \ /\n- resource_type: ceph.smb.share\n cluster_id: modtest1\n share_id:\ \ share2\n cephfs:\n volume: cephfs\n subvolumegroup: smb\n subvolume:\ \ sv2\n path: /\n# --- End Embedded YAML\n" - cephadm.wait_for_service: service: smb.modtest1 - cephadm.exec: host.b: - sleep 30 - '{{ctx.samba_client_container_cmd|join('' '')}} smbclient -U DOMAIN1\\ckent%1115Rose. //{{''host.a''|role_to_remote|attr(''ip_address'')}}/share1 -c ls' - '{{ctx.samba_client_container_cmd|join('' '')}} smbclient -U DOMAIN1\\ckent%1115Rose. //{{''host.a''|role_to_remote|attr(''ip_address'')}}/share2 -c ls' - cephadm.shell: host.a: - cmd: ceph smb apply -i - stdin: "# --- Begin Embedded YAML\n- resource_type: ceph.smb.cluster\n cluster_id:\ \ modtest1\n intent: removed\n- resource_type: ceph.smb.join.auth\n auth_id:\ \ join1-admin\n intent: removed\n- resource_type: ceph.smb.share\n cluster_id:\ \ modtest1\n share_id: share1\n intent: removed\n- resource_type: ceph.smb.share\n\ \ cluster_id: modtest1\n share_id: share2\n intent: removed\n# --- End\ \ Embedded YAML\n" - cephadm.wait_for_service_not_present: service: smb.modtest1 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-25_11:39:08 tube: smithi user: phlogistonjohn verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-25T19:38:56.673 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_b48599b46c4ce29785962a8af5ddf30f6f38e57f/qa; will attempt to use it 2024-04-25T19:38:56.674 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_b48599b46c4ce29785962a8af5ddf30f6f38e57f/qa/tasks 2024-04-25T19:38:56.674 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-25T19:38:56.676 INFO:teuthology.task.internal:Checking packages... 2024-04-25T19:38:56.704 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'b48599b46c4ce29785962a8af5ddf30f6f38e57f' 2024-04-25T19:38:56.704 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-25T19:38:56.704 INFO:teuthology.packaging:ref: None 2024-04-25T19:38:56.704 INFO:teuthology.packaging:tag: None 2024-04-25T19:38:56.704 INFO:teuthology.packaging:branch: wip-phlogistonjohn-testing-2024-04-24-1356 2024-04-25T19:38:56.704 INFO:teuthology.packaging:sha1: b48599b46c4ce29785962a8af5ddf30f6f38e57f 2024-04-25T19:38:56.704 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-phlogistonjohn-testing-2024-04-24-1356 2024-04-25T19:38:56.848 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3290-gb48599b4-1jammy 2024-04-25T19:38:56.850 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-25T19:38:56.858 INFO:teuthology.task.internal:no buildpackages task found 2024-04-25T19:38:56.858 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-25T19:38:56.866 INFO:teuthology.task.internal:Saving configuration 2024-04-25T19:38:56.878 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-25T19:38:56.885 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T19:38:56.911 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi132.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/phlogistonjohn-2024-04-25_11:39:08-orch:cephadm-wip-phlogistonjohn-testing-2024-04-24-1356-distro-default-smithi/7673419', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 19:32:55.755509', 'locked_by': 'scheduled_phlogistonjohn@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIO4hJxGfxe1SdKFwlZCxYFZAq7bkZ6gUc1YepYiYyr4t'} 2024-04-25T19:38:56.935 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/phlogistonjohn-2024-04-25_11:39:08-orch:cephadm-wip-phlogistonjohn-testing-2024-04-24-1356-distro-default-smithi/7673419', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 19:32:55.757167', 'locked_by': 'scheduled_phlogistonjohn@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINC1lbQ4n0UuZqotikxoCawogn4wFC7m3YKVO3NWCUYR'} 2024-04-25T19:38:56.935 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-25T19:38:56.942 INFO:teuthology.task.internal:roles: ubuntu@smithi132.front.sepia.ceph.com - ['host.a', 'mon.a', 'mgr.x', 'osd.0', 'osd.1', 'client.0'] 2024-04-25T19:38:56.942 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['host.b', 'cephadm.exclude'] 2024-04-25T19:38:56.942 INFO:teuthology.run_tasks:Running task console_log... 2024-04-25T19:38:57.016 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1b6ab42280>, signals=[15]) 2024-04-25T19:38:57.016 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-25T19:38:57.064 INFO:teuthology.task.internal:Opening connections... 2024-04-25T19:38:57.064 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi132.front.sepia.ceph.com 2024-04-25T19:38:57.066 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T19:38:57.145 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi150.front.sepia.ceph.com 2024-04-25T19:38:57.146 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T19:38:57.227 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-25T19:38:57.233 DEBUG:teuthology.orchestra.run.smithi132:> uname -m 2024-04-25T19:38:57.239 INFO:teuthology.orchestra.run.smithi132.stdout:x86_64 2024-04-25T19:38:57.239 DEBUG:teuthology.orchestra.run.smithi132:> cat /etc/os-release 2024-04-25T19:38:57.285 INFO:teuthology.orchestra.run.smithi132.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-25T19:38:57.285 INFO:teuthology.orchestra.run.smithi132.stdout:NAME="Ubuntu" 2024-04-25T19:38:57.285 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION_ID="22.04" 2024-04-25T19:38:57.285 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-25T19:38:57.286 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION_CODENAME=jammy 2024-04-25T19:38:57.286 INFO:teuthology.orchestra.run.smithi132.stdout:ID=ubuntu 2024-04-25T19:38:57.286 INFO:teuthology.orchestra.run.smithi132.stdout:ID_LIKE=debian 2024-04-25T19:38:57.286 INFO:teuthology.orchestra.run.smithi132.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-25T19:38:57.286 INFO:teuthology.orchestra.run.smithi132.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-25T19:38:57.286 INFO:teuthology.orchestra.run.smithi132.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-25T19:38:57.286 INFO:teuthology.orchestra.run.smithi132.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-25T19:38:57.286 INFO:teuthology.orchestra.run.smithi132.stdout:UBUNTU_CODENAME=jammy 2024-04-25T19:38:57.287 INFO:teuthology.lock.ops:Updating smithi132.front.sepia.ceph.com on lock server 2024-04-25T19:38:57.309 DEBUG:teuthology.orchestra.run.smithi150:> uname -m 2024-04-25T19:38:57.315 INFO:teuthology.orchestra.run.smithi150.stdout:x86_64 2024-04-25T19:38:57.315 DEBUG:teuthology.orchestra.run.smithi150:> cat /etc/os-release 2024-04-25T19:38:57.362 INFO:teuthology.orchestra.run.smithi150.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-25T19:38:57.362 INFO:teuthology.orchestra.run.smithi150.stdout:NAME="Ubuntu" 2024-04-25T19:38:57.362 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION_ID="22.04" 2024-04-25T19:38:57.363 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-25T19:38:57.363 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION_CODENAME=jammy 2024-04-25T19:38:57.363 INFO:teuthology.orchestra.run.smithi150.stdout:ID=ubuntu 2024-04-25T19:38:57.363 INFO:teuthology.orchestra.run.smithi150.stdout:ID_LIKE=debian 2024-04-25T19:38:57.363 INFO:teuthology.orchestra.run.smithi150.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-25T19:38:57.363 INFO:teuthology.orchestra.run.smithi150.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-25T19:38:57.363 INFO:teuthology.orchestra.run.smithi150.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-25T19:38:57.363 INFO:teuthology.orchestra.run.smithi150.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-25T19:38:57.363 INFO:teuthology.orchestra.run.smithi150.stdout:UBUNTU_CODENAME=jammy 2024-04-25T19:38:57.363 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server 2024-04-25T19:38:57.418 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-25T19:38:57.427 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-25T19:38:57.435 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-25T19:38:57.435 DEBUG:teuthology.orchestra.run.smithi132:> test '!' -e /home/ubuntu/cephtest 2024-04-25T19:38:57.437 DEBUG:teuthology.orchestra.run.smithi150:> test '!' -e /home/ubuntu/cephtest 2024-04-25T19:38:57.442 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-25T19:38:57.450 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-25T19:38:57.450 DEBUG:teuthology.orchestra.run.smithi132:> test -z $(ls -A /var/lib/ceph) 2024-04-25T19:38:57.483 DEBUG:teuthology.orchestra.run.smithi150:> test -z $(ls -A /var/lib/ceph) 2024-04-25T19:38:57.506 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-25T19:38:57.566 INFO:teuthology.run_tasks:Running task kernel... 2024-04-25T19:38:57.581 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-25T19:38:57.582 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-25T19:38:57.582 DEBUG:teuthology.orchestra.run.smithi132:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T19:38:57.583 DEBUG:teuthology.orchestra.run.smithi150:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T19:38:57.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:38:57.587 DEBUG:teuthology.orchestra.run.smithi132:> uname -r 2024-04-25T19:38:57.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:38:57.588 DEBUG:teuthology.orchestra.run.smithi150:> uname -r 2024-04-25T19:38:57.633 INFO:teuthology.orchestra.run.smithi132.stdout:5.15.0-56-generic 2024-04-25T19:38:57.634 INFO:teuthology.task.kernel:Running kernel on smithi132: 5.15.0-56-generic 2024-04-25T19:38:57.634 DEBUG:teuthology.orchestra.run.smithi132:> sudo apt-get clean 2024-04-25T19:38:57.635 INFO:teuthology.orchestra.run.smithi150.stdout:5.15.0-56-generic 2024-04-25T19:38:57.635 INFO:teuthology.task.kernel:Running kernel on smithi150: 5.15.0-56-generic 2024-04-25T19:38:57.635 DEBUG:teuthology.orchestra.run.smithi150:> sudo apt-get clean 2024-04-25T19:38:57.776 DEBUG:teuthology.orchestra.run.smithi150:> sudo apt-get update 2024-04-25T19:38:57.803 DEBUG:teuthology.orchestra.run.smithi132:> sudo apt-get update 2024-04-25T19:38:57.918 INFO:teuthology.orchestra.run.smithi150.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-25T19:38:57.981 INFO:teuthology.orchestra.run.smithi132.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-25T19:38:57.981 INFO:teuthology.orchestra.run.smithi132.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-25T19:38:57.981 INFO:teuthology.orchestra.run.smithi132.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-25T19:38:58.070 INFO:teuthology.orchestra.run.smithi132.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-25T19:38:58.151 INFO:teuthology.orchestra.run.smithi150.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,394 kB] 2024-04-25T19:38:58.226 INFO:teuthology.orchestra.run.smithi132.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [452 kB] 2024-04-25T19:38:58.252 INFO:teuthology.orchestra.run.smithi150.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-25T19:38:58.270 INFO:teuthology.orchestra.run.smithi150.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [452 kB] 2024-04-25T19:38:58.275 INFO:teuthology.orchestra.run.smithi150.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-04-25T19:38:58.278 INFO:teuthology.orchestra.run.smithi150.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-25T19:38:58.278 INFO:teuthology.orchestra.run.smithi150.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-25T19:38:58.279 INFO:teuthology.orchestra.run.smithi150.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,773 kB] 2024-04-25T19:38:58.305 INFO:teuthology.orchestra.run.smithi150.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [300 kB] 2024-04-25T19:38:58.308 INFO:teuthology.orchestra.run.smithi150.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-25T19:38:58.308 INFO:teuthology.orchestra.run.smithi150.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [601 kB] 2024-04-25T19:38:58.312 INFO:teuthology.orchestra.run.smithi132.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,394 kB] 2024-04-25T19:38:58.312 INFO:teuthology.orchestra.run.smithi132.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [618 kB] 2024-04-25T19:38:58.315 INFO:teuthology.orchestra.run.smithi150.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-25T19:38:58.340 INFO:teuthology.orchestra.run.smithi150.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-25T19:38:58.341 INFO:teuthology.orchestra.run.smithi150.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-25T19:38:58.342 INFO:teuthology.orchestra.run.smithi150.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-25T19:38:58.342 INFO:teuthology.orchestra.run.smithi150.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-25T19:38:58.343 INFO:teuthology.orchestra.run.smithi150.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-25T19:38:58.343 INFO:teuthology.orchestra.run.smithi150.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-25T19:38:58.349 INFO:teuthology.orchestra.run.smithi150.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-25T19:38:58.358 INFO:teuthology.orchestra.run.smithi132.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-04-25T19:38:58.360 INFO:teuthology.orchestra.run.smithi132.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-25T19:38:58.360 INFO:teuthology.orchestra.run.smithi132.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,773 kB] 2024-04-25T19:38:58.383 INFO:teuthology.orchestra.run.smithi132.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-25T19:38:58.383 INFO:teuthology.orchestra.run.smithi132.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [300 kB] 2024-04-25T19:38:58.387 INFO:teuthology.orchestra.run.smithi132.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-25T19:38:58.387 INFO:teuthology.orchestra.run.smithi132.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [601 kB] 2024-04-25T19:38:58.393 INFO:teuthology.orchestra.run.smithi132.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-25T19:38:58.394 INFO:teuthology.orchestra.run.smithi132.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,610 kB] 2024-04-25T19:38:58.402 INFO:teuthology.orchestra.run.smithi132.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-25T19:38:58.404 INFO:teuthology.orchestra.run.smithi132.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-25T19:38:58.408 INFO:teuthology.orchestra.run.smithi132.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-25T19:38:58.440 INFO:teuthology.orchestra.run.smithi132.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB] 2024-04-25T19:38:58.441 INFO:teuthology.orchestra.run.smithi132.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-25T19:38:58.441 INFO:teuthology.orchestra.run.smithi132.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-25T19:38:58.441 INFO:teuthology.orchestra.run.smithi132.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-25T19:38:58.442 INFO:teuthology.orchestra.run.smithi132.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-25T19:38:58.443 INFO:teuthology.orchestra.run.smithi132.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,830 kB] 2024-04-25T19:38:58.462 INFO:teuthology.orchestra.run.smithi132.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-25T19:38:58.463 INFO:teuthology.orchestra.run.smithi132.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [311 kB] 2024-04-25T19:38:58.465 INFO:teuthology.orchestra.run.smithi132.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-25T19:38:58.465 INFO:teuthology.orchestra.run.smithi132.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,070 kB] 2024-04-25T19:38:58.476 INFO:teuthology.orchestra.run.smithi132.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [700 kB] 2024-04-25T19:38:58.483 INFO:teuthology.orchestra.run.smithi132.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-25T19:38:58.485 INFO:teuthology.orchestra.run.smithi132.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-25T19:38:58.488 INFO:teuthology.orchestra.run.smithi132.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-25T19:38:58.488 INFO:teuthology.orchestra.run.smithi132.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-25T19:38:58.488 INFO:teuthology.orchestra.run.smithi132.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-25T19:38:58.489 INFO:teuthology.orchestra.run.smithi132.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-25T19:38:58.506 INFO:teuthology.orchestra.run.smithi132.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-25T19:38:58.507 INFO:teuthology.orchestra.run.smithi132.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB] 2024-04-25T19:38:58.508 INFO:teuthology.orchestra.run.smithi132.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-25T19:38:58.508 INFO:teuthology.orchestra.run.smithi132.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-25T19:38:58.508 INFO:teuthology.orchestra.run.smithi132.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB] 2024-04-25T19:38:58.509 INFO:teuthology.orchestra.run.smithi132.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.0 kB] 2024-04-25T19:38:58.529 INFO:teuthology.orchestra.run.smithi132.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-25T19:38:58.529 INFO:teuthology.orchestra.run.smithi132.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-25T19:38:58.725 INFO:teuthology.orchestra.run.smithi150.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-25T19:38:58.864 INFO:teuthology.orchestra.run.smithi150.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [618 kB] 2024-04-25T19:39:00.058 INFO:teuthology.orchestra.run.smithi150.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,610 kB] 2024-04-25T19:39:04.164 INFO:teuthology.orchestra.run.smithi132.stdout:Fetched 13.2 MB in 2s (7,054 kB/s) 2024-04-25T19:39:05.532 INFO:teuthology.orchestra.run.smithi132.stdout:Reading package lists... 2024-04-25T19:39:05.549 DEBUG:teuthology.orchestra.run.smithi132:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-25T19:39:05.608 INFO:teuthology.orchestra.run.smithi132.stdout:Reading package lists... 2024-04-25T19:39:05.809 INFO:teuthology.orchestra.run.smithi132.stdout:Building dependency tree... 2024-04-25T19:39:05.810 INFO:teuthology.orchestra.run.smithi132.stdout:Reading state information... 2024-04-25T19:39:05.985 INFO:teuthology.orchestra.run.smithi132.stdout:The following packages were automatically installed and are no longer required: 2024-04-25T19:39:05.985 INFO:teuthology.orchestra.run.smithi132.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-25T19:39:05.985 INFO:teuthology.orchestra.run.smithi132.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-25T19:39:05.986 INFO:teuthology.orchestra.run.smithi132.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-25T19:39:05.986 INFO:teuthology.orchestra.run.smithi132.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-25T19:39:05.986 INFO:teuthology.orchestra.run.smithi132.stdout: python2.7-minimal 2024-04-25T19:39:05.986 INFO:teuthology.orchestra.run.smithi132.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-25T19:39:05.987 INFO:teuthology.orchestra.run.smithi132.stdout:The following additional packages will be installed: 2024-04-25T19:39:05.987 INFO:teuthology.orchestra.run.smithi132.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-25T19:39:05.987 INFO:teuthology.orchestra.run.smithi132.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-25T19:39:05.988 INFO:teuthology.orchestra.run.smithi132.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-25T19:39:05.989 INFO:teuthology.orchestra.run.smithi132.stdout:Suggested packages: 2024-04-25T19:39:05.989 INFO:teuthology.orchestra.run.smithi132.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-25T19:39:05.989 INFO:teuthology.orchestra.run.smithi132.stdout:Recommended packages: 2024-04-25T19:39:05.989 INFO:teuthology.orchestra.run.smithi132.stdout: thermald 2024-04-25T19:39:06.046 INFO:teuthology.orchestra.run.smithi132.stdout:The following NEW packages will be installed: 2024-04-25T19:39:06.046 INFO:teuthology.orchestra.run.smithi132.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-25T19:39:06.046 INFO:teuthology.orchestra.run.smithi132.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-25T19:39:06.047 INFO:teuthology.orchestra.run.smithi132.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-25T19:39:06.048 INFO:teuthology.orchestra.run.smithi132.stdout:The following packages will be upgraded: 2024-04-25T19:39:06.048 INFO:teuthology.orchestra.run.smithi132.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-25T19:39:06.106 INFO:teuthology.orchestra.run.smithi132.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-25T19:39:06.106 INFO:teuthology.orchestra.run.smithi132.stdout:Need to get 113 MB of archives. 2024-04-25T19:39:06.106 INFO:teuthology.orchestra.run.smithi132.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-25T19:39:06.106 INFO:teuthology.orchestra.run.smithi132.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-25T19:39:06.544 INFO:teuthology.orchestra.run.smithi132.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-25T19:39:06.751 INFO:teuthology.orchestra.run.smithi132.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-25T19:39:06.816 INFO:teuthology.orchestra.run.smithi150.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB] 2024-04-25T19:39:07.893 INFO:teuthology.orchestra.run.smithi132.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-25T19:39:07.893 INFO:teuthology.orchestra.run.smithi132.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-25T19:39:07.893 INFO:teuthology.orchestra.run.smithi132.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-25T19:39:08.005 INFO:teuthology.orchestra.run.smithi150.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-25T19:39:08.043 INFO:teuthology.orchestra.run.smithi132.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-25T19:39:08.054 INFO:teuthology.orchestra.run.smithi150.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-25T19:39:08.116 INFO:teuthology.orchestra.run.smithi132.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-25T19:39:08.141 INFO:teuthology.orchestra.run.smithi150.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,830 kB] 2024-04-25T19:39:08.526 INFO:teuthology.orchestra.run.smithi132.stdout:Fetched 113 MB in 2s (55.0 MB/s) 2024-04-25T19:39:08.710 INFO:teuthology.orchestra.run.smithi132.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-25T19:39:10.758 INFO:teuthology.orchestra.run.smithi132.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-25T19:39:10.762 INFO:teuthology.orchestra.run.smithi132.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T19:39:10.949 INFO:teuthology.orchestra.run.smithi132.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T19:39:13.101 INFO:teuthology.orchestra.run.smithi132.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-25T19:39:13.122 INFO:teuthology.orchestra.run.smithi132.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T19:39:13.200 INFO:teuthology.orchestra.run.smithi132.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T19:39:13.721 INFO:teuthology.orchestra.run.smithi132.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-25T19:39:13.741 INFO:teuthology.orchestra.run.smithi132.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T19:39:13.779 INFO:teuthology.orchestra.run.smithi132.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T19:39:15.606 INFO:teuthology.orchestra.run.smithi150.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [311 kB] 2024-04-25T19:39:16.291 INFO:teuthology.orchestra.run.smithi150.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-25T19:39:16.779 INFO:teuthology.orchestra.run.smithi150.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,070 kB] 2024-04-25T19:39:17.686 INFO:teuthology.orchestra.run.smithi150.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [700 kB] 2024-04-25T19:39:18.048 INFO:teuthology.orchestra.run.smithi150.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-25T19:39:18.152 INFO:teuthology.orchestra.run.smithi150.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-25T19:39:18.155 INFO:teuthology.orchestra.run.smithi150.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-25T19:39:18.156 INFO:teuthology.orchestra.run.smithi150.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-25T19:39:18.225 INFO:teuthology.orchestra.run.smithi150.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-25T19:39:18.226 INFO:teuthology.orchestra.run.smithi150.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-25T19:39:18.226 INFO:teuthology.orchestra.run.smithi150.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-25T19:39:18.239 INFO:teuthology.orchestra.run.smithi150.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB] 2024-04-25T19:39:18.249 INFO:teuthology.orchestra.run.smithi150.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-25T19:39:18.940 INFO:teuthology.orchestra.run.smithi150.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-25T19:39:19.024 INFO:teuthology.orchestra.run.smithi150.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB] 2024-04-25T19:39:19.127 INFO:teuthology.orchestra.run.smithi150.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.0 kB] 2024-04-25T19:39:19.162 INFO:teuthology.orchestra.run.smithi150.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-25T19:39:19.210 INFO:teuthology.orchestra.run.smithi150.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-25T19:39:21.056 INFO:teuthology.orchestra.run.smithi132.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-25T19:39:21.164 INFO:teuthology.orchestra.run.smithi132.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-25T19:39:21.497 INFO:teuthology.orchestra.run.smithi132.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-25T19:39:21.616 INFO:teuthology.orchestra.run.smithi132.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-25T19:39:21.904 INFO:teuthology.orchestra.run.smithi132.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-25T19:39:21.926 INFO:teuthology.orchestra.run.smithi132.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-25T19:39:21.979 INFO:teuthology.orchestra.run.smithi132.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-25T19:39:23.600 INFO:teuthology.orchestra.run.smithi150.stdout:Fetched 13.2 MB in 21s (620 kB/s) 2024-04-25T19:39:25.081 INFO:teuthology.orchestra.run.smithi150.stdout:Reading package lists... 2024-04-25T19:39:25.099 DEBUG:teuthology.orchestra.run.smithi150:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-25T19:39:25.154 INFO:teuthology.orchestra.run.smithi150.stdout:Reading package lists... 2024-04-25T19:39:25.355 INFO:teuthology.orchestra.run.smithi150.stdout:Building dependency tree... 2024-04-25T19:39:25.355 INFO:teuthology.orchestra.run.smithi150.stdout:Reading state information... 2024-04-25T19:39:25.518 INFO:teuthology.orchestra.run.smithi150.stdout:The following packages were automatically installed and are no longer required: 2024-04-25T19:39:25.519 INFO:teuthology.orchestra.run.smithi150.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-25T19:39:25.519 INFO:teuthology.orchestra.run.smithi150.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-25T19:39:25.519 INFO:teuthology.orchestra.run.smithi150.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-25T19:39:25.519 INFO:teuthology.orchestra.run.smithi150.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-25T19:39:25.519 INFO:teuthology.orchestra.run.smithi150.stdout: python2.7-minimal 2024-04-25T19:39:25.519 INFO:teuthology.orchestra.run.smithi150.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-25T19:39:25.520 INFO:teuthology.orchestra.run.smithi150.stdout:The following additional packages will be installed: 2024-04-25T19:39:25.520 INFO:teuthology.orchestra.run.smithi150.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-25T19:39:25.521 INFO:teuthology.orchestra.run.smithi150.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-25T19:39:25.521 INFO:teuthology.orchestra.run.smithi150.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-25T19:39:25.522 INFO:teuthology.orchestra.run.smithi150.stdout:Suggested packages: 2024-04-25T19:39:25.522 INFO:teuthology.orchestra.run.smithi150.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-25T19:39:25.522 INFO:teuthology.orchestra.run.smithi150.stdout:Recommended packages: 2024-04-25T19:39:25.522 INFO:teuthology.orchestra.run.smithi150.stdout: thermald 2024-04-25T19:39:25.579 INFO:teuthology.orchestra.run.smithi150.stdout:The following NEW packages will be installed: 2024-04-25T19:39:25.579 INFO:teuthology.orchestra.run.smithi150.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-25T19:39:25.579 INFO:teuthology.orchestra.run.smithi150.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-25T19:39:25.580 INFO:teuthology.orchestra.run.smithi150.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-25T19:39:25.581 INFO:teuthology.orchestra.run.smithi150.stdout:The following packages will be upgraded: 2024-04-25T19:39:25.581 INFO:teuthology.orchestra.run.smithi150.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-25T19:39:25.639 INFO:teuthology.orchestra.run.smithi150.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-25T19:39:25.639 INFO:teuthology.orchestra.run.smithi150.stdout:Need to get 113 MB of archives. 2024-04-25T19:39:25.639 INFO:teuthology.orchestra.run.smithi150.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-25T19:39:25.639 INFO:teuthology.orchestra.run.smithi150.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-25T19:39:26.202 INFO:teuthology.orchestra.run.smithi150.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-25T19:39:26.365 INFO:teuthology.orchestra.run.smithi150.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-25T19:39:27.604 INFO:teuthology.orchestra.run.smithi150.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-25T19:39:27.604 INFO:teuthology.orchestra.run.smithi150.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-25T19:39:27.604 INFO:teuthology.orchestra.run.smithi150.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-25T19:39:27.744 INFO:teuthology.orchestra.run.smithi150.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-25T19:39:27.850 INFO:teuthology.orchestra.run.smithi132.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-25T19:39:27.852 INFO:teuthology.orchestra.run.smithi150.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-25T19:39:27.885 INFO:teuthology.orchestra.run.smithi132.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T19:39:27.924 INFO:teuthology.orchestra.run.smithi132.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T19:39:28.191 INFO:teuthology.orchestra.run.smithi150.stdout:Fetched 113 MB in 2s (50.0 MB/s) 2024-04-25T19:39:28.454 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-25T19:39:30.484 INFO:teuthology.orchestra.run.smithi150.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-25T19:39:30.488 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T19:39:30.602 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T19:39:30.622 INFO:teuthology.orchestra.run.smithi132.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-25T19:39:30.766 INFO:teuthology.orchestra.run.smithi132.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-25T19:39:31.145 INFO:teuthology.orchestra.run.smithi132.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-25T19:39:31.241 INFO:teuthology.orchestra.run.smithi132.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T19:39:31.343 INFO:teuthology.orchestra.run.smithi132.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-25T19:39:31.563 INFO:teuthology.orchestra.run.smithi132.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T19:39:32.728 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-25T19:39:32.749 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T19:39:32.835 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T19:39:32.858 INFO:teuthology.orchestra.run.smithi132.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-25T19:39:32.858 INFO:teuthology.orchestra.run.smithi132.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-25T19:39:32.858 INFO:teuthology.orchestra.run.smithi132.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-25T19:39:32.858 INFO:teuthology.orchestra.run.smithi132.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-25T19:39:33.048 INFO:teuthology.orchestra.run.smithi132.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T19:39:33.307 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-25T19:39:33.327 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T19:39:33.364 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T19:39:34.701 INFO:teuthology.orchestra.run.smithi132.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T19:39:36.051 INFO:teuthology.orchestra.run.smithi132.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-25T19:39:36.177 INFO:teuthology.orchestra.run.smithi132.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-25T19:39:36.337 INFO:teuthology.orchestra.run.smithi132.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T19:39:36.408 INFO:teuthology.orchestra.run.smithi132.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-25T19:39:36.409 INFO:teuthology.orchestra.run.smithi132.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-25T19:39:40.741 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-25T19:39:40.843 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-25T19:39:41.176 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-25T19:39:41.296 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-25T19:39:41.624 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-25T19:39:41.646 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-25T19:39:41.691 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-25T19:39:47.987 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-25T19:39:48.024 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T19:39:48.053 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T19:39:50.925 INFO:teuthology.orchestra.run.smithi132.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-25T19:39:50.925 INFO:teuthology.orchestra.run.smithi132.stdout:Sourcing file `/etc/default/grub' 2024-04-25T19:39:50.949 INFO:teuthology.orchestra.run.smithi132.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-25T19:39:50.964 INFO:teuthology.orchestra.run.smithi132.stdout:Generating grub configuration file ... 2024-04-25T19:39:51.114 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-25T19:39:51.236 INFO:teuthology.orchestra.run.smithi132.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-25T19:39:51.250 INFO:teuthology.orchestra.run.smithi132.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-25T19:39:51.257 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-25T19:39:51.547 INFO:teuthology.orchestra.run.smithi132.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-25T19:39:51.553 INFO:teuthology.orchestra.run.smithi132.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-25T19:39:51.615 INFO:teuthology.orchestra.run.smithi132.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-25T19:39:51.621 INFO:teuthology.orchestra.run.smithi132.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-25T19:39:51.633 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-25T19:39:51.737 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T19:39:51.798 INFO:teuthology.orchestra.run.smithi132.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-25T19:39:51.798 INFO:teuthology.orchestra.run.smithi132.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-25T19:39:51.798 INFO:teuthology.orchestra.run.smithi132.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-25T19:39:51.848 INFO:teuthology.orchestra.run.smithi132.stdout:done 2024-04-25T19:39:51.863 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-25T19:39:52.017 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T19:39:52.321 DEBUG:teuthology.orchestra.run.smithi132:> dpkg -s linux-image-generic 2024-04-25T19:39:52.342 INFO:teuthology.orchestra.run.smithi132.stdout:Package: linux-image-generic 2024-04-25T19:39:52.342 INFO:teuthology.orchestra.run.smithi132.stdout:Status: install ok installed 2024-04-25T19:39:52.342 INFO:teuthology.orchestra.run.smithi132.stdout:Priority: optional 2024-04-25T19:39:52.342 INFO:teuthology.orchestra.run.smithi132.stdout:Section: kernel 2024-04-25T19:39:52.342 INFO:teuthology.orchestra.run.smithi132.stdout:Installed-Size: 21 2024-04-25T19:39:52.342 INFO:teuthology.orchestra.run.smithi132.stdout:Maintainer: Ubuntu Kernel Team 2024-04-25T19:39:52.343 INFO:teuthology.orchestra.run.smithi132.stdout:Architecture: amd64 2024-04-25T19:39:52.343 INFO:teuthology.orchestra.run.smithi132.stdout:Source: linux-meta 2024-04-25T19:39:52.343 INFO:teuthology.orchestra.run.smithi132.stdout:Version: 5.15.0.105.102 2024-04-25T19:39:52.343 INFO:teuthology.orchestra.run.smithi132.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-25T19:39:52.343 INFO:teuthology.orchestra.run.smithi132.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-25T19:39:52.343 INFO:teuthology.orchestra.run.smithi132.stdout:Recommends: thermald 2024-04-25T19:39:52.343 INFO:teuthology.orchestra.run.smithi132.stdout:Description: Generic Linux kernel image 2024-04-25T19:39:52.343 INFO:teuthology.orchestra.run.smithi132.stdout: This package will always depend on the latest generic kernel image 2024-04-25T19:39:52.343 INFO:teuthology.orchestra.run.smithi132.stdout: available. 2024-04-25T19:39:52.343 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-25T19:39:52.343 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-25T19:39:52.344 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-25T19:39:52.344 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-04-25T19:39:52.344 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi132.front.sepia.ceph.com, path=None, version=distro) 2024-04-25T19:39:52.344 DEBUG:teuthology.orchestra.run.smithi132:> sudo apt-get clean 2024-04-25T19:39:52.565 DEBUG:teuthology.orchestra.run.smithi132:> sudo apt-get update 2024-04-25T19:39:52.691 INFO:teuthology.orchestra.run.smithi132.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-25T19:39:52.704 INFO:teuthology.orchestra.run.smithi132.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-25T19:39:52.725 INFO:teuthology.orchestra.run.smithi132.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-25T19:39:53.327 INFO:teuthology.orchestra.run.smithi150.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-25T19:39:53.327 INFO:teuthology.orchestra.run.smithi150.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-25T19:39:53.327 INFO:teuthology.orchestra.run.smithi150.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-25T19:39:53.327 INFO:teuthology.orchestra.run.smithi150.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-25T19:39:53.502 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T19:39:53.810 INFO:teuthology.orchestra.run.smithi132.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-25T19:39:54.905 INFO:teuthology.orchestra.run.smithi132.stdout:Reading package lists... 2024-04-25T19:39:54.923 DEBUG:teuthology.orchestra.run.smithi132:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-25T19:39:54.946 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T19:39:54.984 INFO:teuthology.orchestra.run.smithi132.stdout:Reading package lists... 2024-04-25T19:39:55.184 INFO:teuthology.orchestra.run.smithi132.stdout:Building dependency tree... 2024-04-25T19:39:55.185 INFO:teuthology.orchestra.run.smithi132.stdout:Reading state information... 2024-04-25T19:39:55.361 INFO:teuthology.orchestra.run.smithi132.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-25T19:39:55.362 INFO:teuthology.orchestra.run.smithi132.stdout:The following packages were automatically installed and are no longer required: 2024-04-25T19:39:55.362 INFO:teuthology.orchestra.run.smithi132.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-25T19:39:55.362 INFO:teuthology.orchestra.run.smithi132.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-25T19:39:55.362 INFO:teuthology.orchestra.run.smithi132.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-25T19:39:55.362 INFO:teuthology.orchestra.run.smithi132.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-25T19:39:55.362 INFO:teuthology.orchestra.run.smithi132.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-25T19:39:55.362 INFO:teuthology.orchestra.run.smithi132.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-25T19:39:55.402 INFO:teuthology.orchestra.run.smithi132.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-25T19:39:55.403 DEBUG:teuthology.orchestra.run.smithi132:> dpkg -s linux-image-generic 2024-04-25T19:39:55.419 INFO:teuthology.orchestra.run.smithi132.stdout:Package: linux-image-generic 2024-04-25T19:39:55.419 INFO:teuthology.orchestra.run.smithi132.stdout:Status: install ok installed 2024-04-25T19:39:55.419 INFO:teuthology.orchestra.run.smithi132.stdout:Priority: optional 2024-04-25T19:39:55.419 INFO:teuthology.orchestra.run.smithi132.stdout:Section: kernel 2024-04-25T19:39:55.419 INFO:teuthology.orchestra.run.smithi132.stdout:Installed-Size: 21 2024-04-25T19:39:55.419 INFO:teuthology.orchestra.run.smithi132.stdout:Maintainer: Ubuntu Kernel Team 2024-04-25T19:39:55.419 INFO:teuthology.orchestra.run.smithi132.stdout:Architecture: amd64 2024-04-25T19:39:55.419 INFO:teuthology.orchestra.run.smithi132.stdout:Source: linux-meta 2024-04-25T19:39:55.419 INFO:teuthology.orchestra.run.smithi132.stdout:Version: 5.15.0.105.102 2024-04-25T19:39:55.420 INFO:teuthology.orchestra.run.smithi132.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-25T19:39:55.420 INFO:teuthology.orchestra.run.smithi132.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-25T19:39:55.420 INFO:teuthology.orchestra.run.smithi132.stdout:Recommends: thermald 2024-04-25T19:39:55.420 INFO:teuthology.orchestra.run.smithi132.stdout:Description: Generic Linux kernel image 2024-04-25T19:39:55.420 INFO:teuthology.orchestra.run.smithi132.stdout: This package will always depend on the latest generic kernel image 2024-04-25T19:39:55.420 INFO:teuthology.orchestra.run.smithi132.stdout: available. 2024-04-25T19:39:55.420 DEBUG:teuthology.orchestra.run.smithi132:> mktemp 2024-04-25T19:39:55.464 INFO:teuthology.orchestra.run.smithi132.stdout:/tmp/tmp.BqBpaKc531 2024-04-25T19:39:55.464 DEBUG:teuthology.orchestra.run.smithi132:> sudo cp /boot/grub/grub.cfg /tmp/tmp.BqBpaKc531 2024-04-25T19:39:55.521 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 0666 /tmp/tmp.BqBpaKc531 2024-04-25T19:39:55.634 DEBUG:teuthology.orchestra.remote:smithi132:/tmp/tmp.BqBpaKc531 is 10KB 2024-04-25T19:39:55.646 DEBUG:teuthology.orchestra.run.smithi132:> rm -fr /tmp/tmp.BqBpaKc531 2024-04-25T19:39:55.652 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-25T19:39:55.705 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-25T19:39:55.705 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-25T19:39:55.705 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-25T19:39:55.796 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-25T19:39:55.796 DEBUG:teuthology.orchestra.run.smithi132:> sudo update-grub 2024-04-25T19:39:56.330 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-25T19:39:56.481 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-25T19:39:56.607 INFO:teuthology.orchestra.run.smithi150.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T19:39:56.675 INFO:teuthology.orchestra.run.smithi150.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-25T19:39:56.675 INFO:teuthology.orchestra.run.smithi150.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-25T19:39:57.215 INFO:teuthology.orchestra.run.smithi132.stderr:Sourcing file `/etc/default/grub' 2024-04-25T19:39:57.217 INFO:teuthology.orchestra.run.smithi132.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-25T19:39:57.220 INFO:teuthology.orchestra.run.smithi132.stderr:Generating grub configuration file ... 2024-04-25T19:39:57.404 INFO:teuthology.orchestra.run.smithi132.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-25T19:39:57.410 INFO:teuthology.orchestra.run.smithi132.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-25T19:39:57.645 INFO:teuthology.orchestra.run.smithi132.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-25T19:39:57.650 INFO:teuthology.orchestra.run.smithi132.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-25T19:39:57.688 INFO:teuthology.orchestra.run.smithi132.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-25T19:39:57.693 INFO:teuthology.orchestra.run.smithi132.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-25T19:39:57.842 INFO:teuthology.orchestra.run.smithi132.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-25T19:39:57.843 INFO:teuthology.orchestra.run.smithi132.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-25T19:39:57.843 INFO:teuthology.orchestra.run.smithi132.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-25T19:39:57.872 INFO:teuthology.orchestra.run.smithi132.stderr:done 2024-04-25T19:39:57.881 DEBUG:teuthology.orchestra.run.smithi132:> sudo shutdown -r now 2024-04-25T19:40:11.406 INFO:teuthology.orchestra.run.smithi150.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-25T19:40:11.407 INFO:teuthology.orchestra.run.smithi150.stdout:Sourcing file `/etc/default/grub' 2024-04-25T19:40:11.435 INFO:teuthology.orchestra.run.smithi150.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-25T19:40:11.463 INFO:teuthology.orchestra.run.smithi150.stdout:Generating grub configuration file ... 2024-04-25T19:40:11.744 INFO:teuthology.orchestra.run.smithi150.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-25T19:40:11.770 INFO:teuthology.orchestra.run.smithi150.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-25T19:40:12.125 INFO:teuthology.orchestra.run.smithi150.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-25T19:40:12.130 INFO:teuthology.orchestra.run.smithi150.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-25T19:40:12.201 INFO:teuthology.orchestra.run.smithi150.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-25T19:40:12.206 INFO:teuthology.orchestra.run.smithi150.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-25T19:40:12.353 INFO:teuthology.orchestra.run.smithi150.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-25T19:40:12.353 INFO:teuthology.orchestra.run.smithi150.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-25T19:40:12.354 INFO:teuthology.orchestra.run.smithi150.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-25T19:40:12.407 INFO:teuthology.orchestra.run.smithi150.stdout:done 2024-04-25T19:40:12.926 DEBUG:teuthology.orchestra.run.smithi150:> dpkg -s linux-image-generic 2024-04-25T19:40:12.950 INFO:teuthology.orchestra.run.smithi150.stdout:Package: linux-image-generic 2024-04-25T19:40:12.950 INFO:teuthology.orchestra.run.smithi150.stdout:Status: install ok installed 2024-04-25T19:40:12.950 INFO:teuthology.orchestra.run.smithi150.stdout:Priority: optional 2024-04-25T19:40:12.950 INFO:teuthology.orchestra.run.smithi150.stdout:Section: kernel 2024-04-25T19:40:12.950 INFO:teuthology.orchestra.run.smithi150.stdout:Installed-Size: 21 2024-04-25T19:40:12.950 INFO:teuthology.orchestra.run.smithi150.stdout:Maintainer: Ubuntu Kernel Team 2024-04-25T19:40:12.950 INFO:teuthology.orchestra.run.smithi150.stdout:Architecture: amd64 2024-04-25T19:40:12.950 INFO:teuthology.orchestra.run.smithi150.stdout:Source: linux-meta 2024-04-25T19:40:12.950 INFO:teuthology.orchestra.run.smithi150.stdout:Version: 5.15.0.105.102 2024-04-25T19:40:12.950 INFO:teuthology.orchestra.run.smithi150.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-25T19:40:12.950 INFO:teuthology.orchestra.run.smithi150.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-25T19:40:12.951 INFO:teuthology.orchestra.run.smithi150.stdout:Recommends: thermald 2024-04-25T19:40:12.951 INFO:teuthology.orchestra.run.smithi150.stdout:Description: Generic Linux kernel image 2024-04-25T19:40:12.951 INFO:teuthology.orchestra.run.smithi150.stdout: This package will always depend on the latest generic kernel image 2024-04-25T19:40:12.951 INFO:teuthology.orchestra.run.smithi150.stdout: available. 2024-04-25T19:40:12.951 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-25T19:40:12.951 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-25T19:40:12.951 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-25T19:40:12.951 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-04-25T19:40:12.951 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi150.front.sepia.ceph.com, path=None, version=distro) 2024-04-25T19:40:12.951 DEBUG:teuthology.orchestra.run.smithi150:> sudo apt-get clean 2024-04-25T19:40:13.140 DEBUG:teuthology.orchestra.run.smithi150:> sudo apt-get update 2024-04-25T19:40:14.057 INFO:teuthology.orchestra.run.smithi150.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-25T19:40:14.433 INFO:teuthology.orchestra.run.smithi150.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-25T19:40:14.513 INFO:teuthology.orchestra.run.smithi150.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-25T19:40:14.596 INFO:teuthology.orchestra.run.smithi150.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-25T19:40:15.731 INFO:teuthology.orchestra.run.smithi150.stdout:Reading package lists... 2024-04-25T19:40:15.749 DEBUG:teuthology.orchestra.run.smithi150:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-25T19:40:15.809 INFO:teuthology.orchestra.run.smithi150.stdout:Reading package lists... 2024-04-25T19:40:16.005 INFO:teuthology.orchestra.run.smithi150.stdout:Building dependency tree... 2024-04-25T19:40:16.005 INFO:teuthology.orchestra.run.smithi150.stdout:Reading state information... 2024-04-25T19:40:16.183 INFO:teuthology.orchestra.run.smithi150.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-25T19:40:16.185 INFO:teuthology.orchestra.run.smithi150.stdout:The following packages were automatically installed and are no longer required: 2024-04-25T19:40:16.185 INFO:teuthology.orchestra.run.smithi150.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-25T19:40:16.185 INFO:teuthology.orchestra.run.smithi150.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-25T19:40:16.185 INFO:teuthology.orchestra.run.smithi150.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-25T19:40:16.186 INFO:teuthology.orchestra.run.smithi150.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-25T19:40:16.186 INFO:teuthology.orchestra.run.smithi150.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-25T19:40:16.186 INFO:teuthology.orchestra.run.smithi150.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-25T19:40:16.224 INFO:teuthology.orchestra.run.smithi150.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-25T19:40:16.225 DEBUG:teuthology.orchestra.run.smithi150:> dpkg -s linux-image-generic 2024-04-25T19:40:16.237 INFO:teuthology.orchestra.run.smithi150.stdout:Package: linux-image-generic 2024-04-25T19:40:16.237 INFO:teuthology.orchestra.run.smithi150.stdout:Status: install ok installed 2024-04-25T19:40:16.238 INFO:teuthology.orchestra.run.smithi150.stdout:Priority: optional 2024-04-25T19:40:16.238 INFO:teuthology.orchestra.run.smithi150.stdout:Section: kernel 2024-04-25T19:40:16.238 INFO:teuthology.orchestra.run.smithi150.stdout:Installed-Size: 21 2024-04-25T19:40:16.238 INFO:teuthology.orchestra.run.smithi150.stdout:Maintainer: Ubuntu Kernel Team 2024-04-25T19:40:16.238 INFO:teuthology.orchestra.run.smithi150.stdout:Architecture: amd64 2024-04-25T19:40:16.238 INFO:teuthology.orchestra.run.smithi150.stdout:Source: linux-meta 2024-04-25T19:40:16.238 INFO:teuthology.orchestra.run.smithi150.stdout:Version: 5.15.0.105.102 2024-04-25T19:40:16.238 INFO:teuthology.orchestra.run.smithi150.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-25T19:40:16.238 INFO:teuthology.orchestra.run.smithi150.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-25T19:40:16.238 INFO:teuthology.orchestra.run.smithi150.stdout:Recommends: thermald 2024-04-25T19:40:16.238 INFO:teuthology.orchestra.run.smithi150.stdout:Description: Generic Linux kernel image 2024-04-25T19:40:16.238 INFO:teuthology.orchestra.run.smithi150.stdout: This package will always depend on the latest generic kernel image 2024-04-25T19:40:16.238 INFO:teuthology.orchestra.run.smithi150.stdout: available. 2024-04-25T19:40:16.239 DEBUG:teuthology.orchestra.run.smithi150:> mktemp 2024-04-25T19:40:16.285 INFO:teuthology.orchestra.run.smithi150.stdout:/tmp/tmp.AvwAB0kUfO 2024-04-25T19:40:16.285 DEBUG:teuthology.orchestra.run.smithi150:> sudo cp /boot/grub/grub.cfg /tmp/tmp.AvwAB0kUfO 2024-04-25T19:40:16.347 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0666 /tmp/tmp.AvwAB0kUfO 2024-04-25T19:40:16.455 DEBUG:teuthology.orchestra.remote:smithi150:/tmp/tmp.AvwAB0kUfO is 10KB 2024-04-25T19:40:16.468 DEBUG:teuthology.orchestra.run.smithi150:> rm -fr /tmp/tmp.AvwAB0kUfO 2024-04-25T19:40:16.473 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-25T19:40:16.525 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T19:40:16.525 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-25T19:40:16.525 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-25T19:40:16.590 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-25T19:40:16.590 DEBUG:teuthology.orchestra.run.smithi150:> sudo update-grub 2024-04-25T19:40:17.982 INFO:teuthology.orchestra.run.smithi150.stderr:Sourcing file `/etc/default/grub' 2024-04-25T19:40:17.984 INFO:teuthology.orchestra.run.smithi150.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-25T19:40:17.987 INFO:teuthology.orchestra.run.smithi150.stderr:Generating grub configuration file ... 2024-04-25T19:40:18.175 INFO:teuthology.orchestra.run.smithi150.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-25T19:40:18.181 INFO:teuthology.orchestra.run.smithi150.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-25T19:40:18.455 INFO:teuthology.orchestra.run.smithi150.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-25T19:40:18.460 INFO:teuthology.orchestra.run.smithi150.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-25T19:40:18.500 INFO:teuthology.orchestra.run.smithi150.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-25T19:40:18.505 INFO:teuthology.orchestra.run.smithi150.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-25T19:40:18.628 INFO:teuthology.orchestra.run.smithi150.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-25T19:40:18.628 INFO:teuthology.orchestra.run.smithi150.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-25T19:40:18.628 INFO:teuthology.orchestra.run.smithi150.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-25T19:40:18.650 INFO:teuthology.orchestra.run.smithi150.stderr:done 2024-04-25T19:40:18.664 DEBUG:teuthology.orchestra.run.smithi150:> sudo shutdown -r now 2024-04-25T19:40:27.890 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-04-25T19:40:27.891 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-04-25T19:40:27.891 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T19:40:46.239 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.132 2024-04-25T19:40:48.669 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-04-25T19:40:48.669 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-04-25T19:40:48.670 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T19:40:55.246 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-04-25T19:40:55.247 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T19:41:07.231 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-04-25T19:41:16.235 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-04-25T19:41:16.236 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T19:41:19.294 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-04-25T19:41:31.306 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-04-25T19:41:31.307 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T19:41:55.270 DEBUG:teuthology.orchestra.remote:timed out 2024-04-25T19:42:07.282 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-04-25T19:42:07.283 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T19:42:07.674 DEBUG:teuthology.orchestra.run.smithi132:> true 2024-04-25T19:42:08.228 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-04-25T19:42:08.228 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.15.0-105-generic"... 2024-04-25T19:42:08.229 DEBUG:teuthology.orchestra.run.smithi132:> uname -r 2024-04-25T19:42:08.275 INFO:teuthology.orchestra.run.smithi132.stdout:5.15.0-105-generic 2024-04-25T19:42:08.275 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-25T19:42:08.275 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-25T19:42:08.276 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-25T19:42:09.276 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-04-25T19:42:09.276 DEBUG:teuthology.orchestra.run.smithi132:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-25T19:42:09.371 INFO:teuthology.orchestra.run.smithi132.stdout:ttyS1 2024-04-25T19:42:09.397 DEBUG:teuthology.parallel:result is None 2024-04-25T19:42:31.310 DEBUG:teuthology.orchestra.remote:timed out 2024-04-25T19:42:46.310 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-04-25T19:42:46.311 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T19:42:46.626 DEBUG:teuthology.orchestra.run.smithi150:> true 2024-04-25T19:42:47.253 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-04-25T19:42:47.253 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.15.0-105-generic"... 2024-04-25T19:42:47.253 DEBUG:teuthology.orchestra.run.smithi150:> uname -r 2024-04-25T19:42:47.299 INFO:teuthology.orchestra.run.smithi150.stdout:5.15.0-105-generic 2024-04-25T19:42:47.299 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-25T19:42:47.299 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-25T19:42:47.299 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-25T19:42:48.299 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-04-25T19:42:48.300 DEBUG:teuthology.orchestra.run.smithi150:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-25T19:42:48.382 INFO:teuthology.orchestra.run.smithi150.stdout:ttyS1 2024-04-25T19:42:48.405 DEBUG:teuthology.parallel:result is None 2024-04-25T19:42:48.405 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-25T19:42:48.412 INFO:teuthology.task.internal:Creating test directory... 2024-04-25T19:42:48.412 DEBUG:teuthology.orchestra.run.smithi132:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T19:42:48.415 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T19:42:48.420 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-25T19:42:48.427 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-25T19:42:48.434 INFO:teuthology.task.internal:Creating archive directory... 2024-04-25T19:42:48.434 DEBUG:teuthology.orchestra.run.smithi132:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T19:42:48.464 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T19:42:48.480 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-25T19:42:48.487 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-25T19:42:48.487 DEBUG:teuthology.orchestra.run.smithi132:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T19:42:48.520 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T19:42:48.536 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T19:42:48.539 INFO:teuthology.orchestra.run.smithi132.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T19:42:48.544 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T19:42:48.548 INFO:teuthology.orchestra.run.smithi132.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T19:42:48.549 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-25T19:42:48.563 INFO:teuthology.task.internal:Configuring sudo... 2024-04-25T19:42:48.563 DEBUG:teuthology.orchestra.run.smithi132:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T19:42:48.591 DEBUG:teuthology.orchestra.run.smithi150:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T19:42:48.614 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-25T19:42:48.628 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-25T19:42:48.629 DEBUG:teuthology.orchestra.run.smithi132:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T19:42:48.648 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T19:42:48.660 DEBUG:teuthology.orchestra.run.smithi132:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T19:42:48.699 DEBUG:teuthology.orchestra.run.smithi132:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T19:42:48.747 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-25T19:42:48.747 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T19:42:48.807 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T19:42:48.814 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T19:42:48.864 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T19:42:48.864 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T19:42:48.924 DEBUG:teuthology.orchestra.run.smithi132:> sudo service rsyslog restart 2024-04-25T19:42:48.926 DEBUG:teuthology.orchestra.run.smithi150:> sudo service rsyslog restart 2024-04-25T19:42:49.001 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-25T19:42:49.009 INFO:teuthology.task.internal:Starting timer... 2024-04-25T19:42:49.009 INFO:teuthology.run_tasks:Running task pcp... 2024-04-25T19:42:49.021 INFO:teuthology.run_tasks:Running task selinux... 2024-04-25T19:42:49.029 DEBUG:teuthology.task.selinux:Excluding smithi132: OS 'ubuntu' does not support SELinux 2024-04-25T19:42:49.029 DEBUG:teuthology.task.selinux:Excluding smithi150: OS 'ubuntu' does not support SELinux 2024-04-25T19:42:49.029 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-25T19:42:49.029 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-04-25T19:42:49.029 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-25T19:42:49.029 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-25T19:42:49.040 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-25T19:42:49.056 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-25T19:42:49.058 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi132.front.sepia.ceph.com,smithi150.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-25T19:50:05.386 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi132.front.sepia.ceph.com'), Remote(name='ubuntu@smithi150.front.sepia.ceph.com')] 2024-04-25T19:50:05.388 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-04-25T19:50:05.391 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T19:50:05.475 DEBUG:teuthology.orchestra.run.smithi132:> true 2024-04-25T19:50:05.555 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-04-25T19:50:05.555 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-04-25T19:50:05.556 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T19:50:05.636 DEBUG:teuthology.orchestra.run.smithi150:> true 2024-04-25T19:50:05.714 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-04-25T19:50:05.714 INFO:teuthology.run_tasks:Running task clock... 2024-04-25T19:50:05.725 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-25T19:50:05.725 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T19:50:05.725 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T19:50:05.728 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T19:50:05.728 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T19:50:05.754 INFO:teuthology.orchestra.run.smithi132.stdout:25 Apr 19:50:05 ntpd[17236]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-25T19:50:05.755 INFO:teuthology.orchestra.run.smithi132.stdout:25 Apr 19:50:05 ntpd[17236]: Command line: ntpd -gq 2024-04-25T19:50:05.755 INFO:teuthology.orchestra.run.smithi132.stdout:25 Apr 19:50:05 ntpd[17236]: ---------------------------------------------------- 2024-04-25T19:50:05.755 INFO:teuthology.orchestra.run.smithi132.stdout:25 Apr 19:50:05 ntpd[17236]: ntp-4 is maintained by Network Time Foundation, 2024-04-25T19:50:05.755 INFO:teuthology.orchestra.run.smithi132.stdout:25 Apr 19:50:05 ntpd[17236]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-25T19:50:05.755 INFO:teuthology.orchestra.run.smithi132.stdout:25 Apr 19:50:05 ntpd[17236]: corporation. Support and training for ntp-4 are 2024-04-25T19:50:05.755 INFO:teuthology.orchestra.run.smithi132.stdout:25 Apr 19:50:05 ntpd[17236]: available at https://www.nwtime.org/support 2024-04-25T19:50:05.755 INFO:teuthology.orchestra.run.smithi132.stdout:25 Apr 19:50:05 ntpd[17236]: ---------------------------------------------------- 2024-04-25T19:50:05.756 INFO:teuthology.orchestra.run.smithi132.stdout:25 Apr 19:50:05 ntpd[17236]: proto: precision = 0.061 usec (-24) 2024-04-25T19:50:05.756 INFO:teuthology.orchestra.run.smithi132.stdout:25 Apr 19:50:05 ntpd[17236]: basedate set to 2022-02-04 2024-04-25T19:50:05.756 INFO:teuthology.orchestra.run.smithi132.stdout:25 Apr 19:50:05 ntpd[17236]: gps base set to 2022-02-06 (week 2196) 2024-04-25T19:50:05.756 INFO:teuthology.orchestra.run.smithi132.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-25T19:50:05.756 INFO:teuthology.orchestra.run.smithi132.stdout:25 Apr 19:50:05 ntpd[17236]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-25T19:50:05.757 INFO:teuthology.orchestra.run.smithi132.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-25T19:50:05.758 INFO:teuthology.orchestra.run.smithi132.stdout:25 Apr 19:50:05 ntpd[17236]: restrict ::: KOD does nothing without LIMITED. 2024-04-25T19:50:05.758 INFO:teuthology.orchestra.run.smithi132.stdout:25 Apr 19:50:05 ntpd[17236]: Listen and drop on 0 v6wildcard [::]:123 2024-04-25T19:50:05.758 INFO:teuthology.orchestra.run.smithi132.stdout:25 Apr 19:50:05 ntpd[17236]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-25T19:50:05.758 INFO:teuthology.orchestra.run.smithi132.stdout:25 Apr 19:50:05 ntpd[17236]: Listen normally on 2 lo 127.0.0.1:123 2024-04-25T19:50:05.758 INFO:teuthology.orchestra.run.smithi132.stdout:25 Apr 19:50:05 ntpd[17236]: Listen normally on 3 enp3s0f1 172.21.15.132:123 2024-04-25T19:50:05.758 INFO:teuthology.orchestra.run.smithi132.stdout:25 Apr 19:50:05 ntpd[17236]: Listen normally on 4 lo [::1]:123 2024-04-25T19:50:05.758 INFO:teuthology.orchestra.run.smithi132.stdout:25 Apr 19:50:05 ntpd[17236]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:773f%5]:123 2024-04-25T19:50:05.758 INFO:teuthology.orchestra.run.smithi132.stdout:25 Apr 19:50:05 ntpd[17236]: Listening on routing socket on fd #22 for interface updates 2024-04-25T19:50:05.785 INFO:teuthology.orchestra.run.smithi150.stdout:25 Apr 19:50:05 ntpd[17204]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-25T19:50:05.785 INFO:teuthology.orchestra.run.smithi150.stdout:25 Apr 19:50:05 ntpd[17204]: Command line: ntpd -gq 2024-04-25T19:50:05.785 INFO:teuthology.orchestra.run.smithi150.stdout:25 Apr 19:50:05 ntpd[17204]: ---------------------------------------------------- 2024-04-25T19:50:05.785 INFO:teuthology.orchestra.run.smithi150.stdout:25 Apr 19:50:05 ntpd[17204]: ntp-4 is maintained by Network Time Foundation, 2024-04-25T19:50:05.786 INFO:teuthology.orchestra.run.smithi150.stdout:25 Apr 19:50:05 ntpd[17204]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-25T19:50:05.786 INFO:teuthology.orchestra.run.smithi150.stdout:25 Apr 19:50:05 ntpd[17204]: corporation. Support and training for ntp-4 are 2024-04-25T19:50:05.786 INFO:teuthology.orchestra.run.smithi150.stdout:25 Apr 19:50:05 ntpd[17204]: available at https://www.nwtime.org/support 2024-04-25T19:50:05.786 INFO:teuthology.orchestra.run.smithi150.stdout:25 Apr 19:50:05 ntpd[17204]: ---------------------------------------------------- 2024-04-25T19:50:05.786 INFO:teuthology.orchestra.run.smithi150.stdout:25 Apr 19:50:05 ntpd[17204]: proto: precision = 0.059 usec (-24) 2024-04-25T19:50:05.786 INFO:teuthology.orchestra.run.smithi150.stdout:25 Apr 19:50:05 ntpd[17204]: basedate set to 2022-02-04 2024-04-25T19:50:05.787 INFO:teuthology.orchestra.run.smithi150.stdout:25 Apr 19:50:05 ntpd[17204]: gps base set to 2022-02-06 (week 2196) 2024-04-25T19:50:05.787 INFO:teuthology.orchestra.run.smithi150.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-25T19:50:05.787 INFO:teuthology.orchestra.run.smithi150.stdout:25 Apr 19:50:05 ntpd[17204]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-25T19:50:05.787 INFO:teuthology.orchestra.run.smithi150.stdout:25 Apr 19:50:05 ntpd[17204]: restrict ::: KOD does nothing without LIMITED. 2024-04-25T19:50:05.788 INFO:teuthology.orchestra.run.smithi150.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-25T19:50:05.788 INFO:teuthology.orchestra.run.smithi150.stdout:25 Apr 19:50:05 ntpd[17204]: Listen and drop on 0 v6wildcard [::]:123 2024-04-25T19:50:05.789 INFO:teuthology.orchestra.run.smithi150.stdout:25 Apr 19:50:05 ntpd[17204]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-25T19:50:05.789 INFO:teuthology.orchestra.run.smithi150.stdout:25 Apr 19:50:05 ntpd[17204]: Listen normally on 2 lo 127.0.0.1:123 2024-04-25T19:50:05.789 INFO:teuthology.orchestra.run.smithi150.stdout:25 Apr 19:50:05 ntpd[17204]: Listen normally on 3 enp3s0f1 172.21.15.150:123 2024-04-25T19:50:05.789 INFO:teuthology.orchestra.run.smithi150.stdout:25 Apr 19:50:05 ntpd[17204]: Listen normally on 4 lo [::1]:123 2024-04-25T19:50:05.789 INFO:teuthology.orchestra.run.smithi150.stdout:25 Apr 19:50:05 ntpd[17204]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6e35%5]:123 2024-04-25T19:50:05.789 INFO:teuthology.orchestra.run.smithi150.stdout:25 Apr 19:50:05 ntpd[17204]: Listening on routing socket on fd #22 for interface updates 2024-04-25T19:50:06.755 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:06 ntpd[17236]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T19:50:06.755 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:06 ntpd[17236]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T19:50:06.755 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:06 ntpd[17236]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T19:50:06.755 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:06 ntpd[17236]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T19:50:06.755 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:06 ntpd[17236]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T19:50:06.755 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:06 ntpd[17236]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T19:50:06.755 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:06 ntpd[17236]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T19:50:06.755 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:06 ntpd[17236]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T19:50:06.787 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:06 ntpd[17204]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T19:50:06.787 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:06 ntpd[17204]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T19:50:06.787 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:06 ntpd[17204]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T19:50:06.788 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:06 ntpd[17204]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T19:50:06.788 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:06 ntpd[17204]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T19:50:06.788 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:06 ntpd[17204]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T19:50:06.788 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:06 ntpd[17204]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T19:50:06.788 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:06 ntpd[17204]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T19:50:06.788 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:06 ntpd[17204]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T19:50:06.788 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:06 ntpd[17204]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T19:50:06.789 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:06 ntpd[17204]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T19:50:06.789 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:06 ntpd[17204]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T19:50:07.754 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:07 ntpd[17236]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T19:50:07.754 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:07 ntpd[17236]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T19:50:07.755 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:07 ntpd[17236]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T19:50:07.755 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:07 ntpd[17236]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T19:50:07.755 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:07 ntpd[17236]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T19:50:07.755 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:07 ntpd[17236]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T19:50:08.755 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:08 ntpd[17236]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T19:50:08.755 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:08 ntpd[17236]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T19:50:08.755 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:08 ntpd[17236]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T19:50:08.755 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:08 ntpd[17236]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T19:50:08.755 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:08 ntpd[17236]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T19:50:08.755 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:08 ntpd[17236]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T19:50:08.755 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:08 ntpd[17236]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T19:50:08.755 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:08 ntpd[17236]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T19:50:08.755 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:08 ntpd[17236]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T19:50:08.756 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:08 ntpd[17236]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T19:50:08.756 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:08 ntpd[17236]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T19:50:08.756 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:08 ntpd[17236]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T19:50:08.787 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:08 ntpd[17204]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T19:50:08.787 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:08 ntpd[17204]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T19:50:08.787 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:08 ntpd[17204]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T19:50:08.787 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:08 ntpd[17204]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T19:50:08.787 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:08 ntpd[17204]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T19:50:08.787 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:08 ntpd[17204]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T19:50:08.788 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:08 ntpd[17204]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T19:50:08.788 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:08 ntpd[17204]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T19:50:08.788 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:08 ntpd[17204]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T19:50:08.788 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:08 ntpd[17204]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T19:50:08.788 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:08 ntpd[17204]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T19:50:08.788 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:08 ntpd[17204]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T19:50:08.788 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:08 ntpd[17204]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T19:50:08.788 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:08 ntpd[17204]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T19:50:08.788 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:08 ntpd[17204]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T19:50:08.788 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:08 ntpd[17204]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T19:50:08.788 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:08 ntpd[17204]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T19:50:08.788 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:08 ntpd[17204]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T19:50:09.754 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:09 ntpd[17236]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T19:50:09.754 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:09 ntpd[17236]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T19:50:09.755 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:09 ntpd[17236]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T19:50:09.755 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:09 ntpd[17236]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T19:50:09.755 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:09 ntpd[17236]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T19:50:09.755 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:09 ntpd[17236]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T19:50:10.755 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:10 ntpd[17236]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T19:50:10.755 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:10 ntpd[17236]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T19:50:10.755 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:10 ntpd[17236]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T19:50:10.755 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:10 ntpd[17236]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T19:50:10.755 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:10 ntpd[17236]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T19:50:10.755 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:10 ntpd[17236]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T19:50:10.756 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:10 ntpd[17236]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T19:50:10.756 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:10 ntpd[17236]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T19:50:10.756 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:10 ntpd[17236]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T19:50:10.756 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:10 ntpd[17236]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T19:50:10.756 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:10 ntpd[17236]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T19:50:10.756 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:10 ntpd[17236]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T19:50:10.787 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:10 ntpd[17204]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T19:50:10.788 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:10 ntpd[17204]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T19:50:10.788 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:10 ntpd[17204]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T19:50:10.788 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:10 ntpd[17204]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T19:50:10.788 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:10 ntpd[17204]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T19:50:10.788 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:10 ntpd[17204]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T19:50:10.788 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:10 ntpd[17204]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T19:50:10.788 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:10 ntpd[17204]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T19:50:10.788 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:10 ntpd[17204]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T19:50:10.788 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:10 ntpd[17204]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T19:50:10.788 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:10 ntpd[17204]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T19:50:10.788 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:10 ntpd[17204]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T19:50:10.789 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:10 ntpd[17204]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T19:50:10.789 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:10 ntpd[17204]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T19:50:10.789 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:10 ntpd[17204]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T19:50:10.789 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:10 ntpd[17204]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T19:50:10.789 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:10 ntpd[17204]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T19:50:10.789 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:10 ntpd[17204]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T19:50:10.789 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:10 ntpd[17204]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T19:50:10.789 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:10 ntpd[17204]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T19:50:11.754 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:11 ntpd[17236]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T19:50:11.754 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:11 ntpd[17236]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T19:50:11.755 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:11 ntpd[17236]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T19:50:11.755 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:11 ntpd[17236]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T19:50:11.755 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:11 ntpd[17236]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T19:50:11.755 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:11 ntpd[17236]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T19:50:12.755 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:12 ntpd[17236]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T19:50:12.755 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:12 ntpd[17236]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T19:50:12.755 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:12 ntpd[17236]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T19:50:12.756 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:12 ntpd[17236]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T19:50:12.756 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:12 ntpd[17236]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T19:50:12.756 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:12 ntpd[17236]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T19:50:12.756 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:12 ntpd[17236]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T19:50:12.756 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:12 ntpd[17236]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T19:50:12.756 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:12 ntpd[17236]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T19:50:12.756 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:12 ntpd[17236]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T19:50:12.756 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:12 ntpd[17236]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T19:50:12.756 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:12 ntpd[17236]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T19:50:12.756 INFO:teuthology.orchestra.run.smithi132.stdout:25 Apr 19:50:12 ntpd[17236]: ntpd: time slew -0.000150 s 2024-04-25T19:50:12.756 INFO:teuthology.orchestra.run.smithi132.stdout:ntpd: time slew -0.000150s 2024-04-25T19:50:12.757 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:12 ntpd[17236]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-25T19:50:12.757 INFO:teuthology.orchestra.run.smithi132.stderr:25 Apr 19:50:12 ntpd[17236]: can't open /var/log/ntpstats/loopstats.20240425: Permission denied 2024-04-25T19:50:12.787 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:12 ntpd[17204]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T19:50:12.788 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:12 ntpd[17204]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T19:50:12.788 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:12 ntpd[17204]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T19:50:12.788 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:12 ntpd[17204]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T19:50:12.788 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:12 ntpd[17204]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T19:50:12.788 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:12 ntpd[17204]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T19:50:12.788 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:12 ntpd[17204]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T19:50:12.788 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:12 ntpd[17204]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T19:50:12.788 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:12 ntpd[17204]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T19:50:12.788 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:12 ntpd[17204]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T19:50:12.789 INFO:teuthology.orchestra.run.smithi150.stdout:25 Apr 19:50:12 ntpd[17204]: ntpd: time slew -0.000904 s 2024-04-25T19:50:12.789 INFO:teuthology.orchestra.run.smithi150.stdout:ntpd: time slew -0.000904s 2024-04-25T19:50:12.789 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:12 ntpd[17204]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-25T19:50:12.789 INFO:teuthology.orchestra.run.smithi150.stderr:25 Apr 19:50:12 ntpd[17204]: can't open /var/log/ntpstats/loopstats.20240425: Permission denied 2024-04-25T19:50:12.803 INFO:teuthology.orchestra.run.smithi132.stdout: remote refid st t when poll reach delay offset jitter 2024-04-25T19:50:12.803 INFO:teuthology.orchestra.run.smithi132.stdout:============================================================================== 2024-04-25T19:50:12.804 INFO:teuthology.orchestra.run.smithi132.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T19:50:12.804 INFO:teuthology.orchestra.run.smithi132.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T19:50:12.804 INFO:teuthology.orchestra.run.smithi132.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T19:50:12.804 INFO:teuthology.orchestra.run.smithi132.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T19:50:12.843 INFO:teuthology.orchestra.run.smithi150.stdout: remote refid st t when poll reach delay offset jitter 2024-04-25T19:50:12.843 INFO:teuthology.orchestra.run.smithi150.stdout:============================================================================== 2024-04-25T19:50:12.843 INFO:teuthology.orchestra.run.smithi150.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T19:50:12.843 INFO:teuthology.orchestra.run.smithi150.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T19:50:12.843 INFO:teuthology.orchestra.run.smithi150.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T19:50:12.843 INFO:teuthology.orchestra.run.smithi150.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T19:50:12.844 INFO:teuthology.run_tasks:Running task cephadm.deploy_samba_ad_dc... 2024-04-25T19:50:12.954 INFO:tasks.cephadm:Testing if podman is available 2024-04-25T19:50:12.954 DEBUG:teuthology.orchestra.run.smithi150:> sudo podman --help 2024-04-25T19:50:12.964 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: podman: command not found 2024-04-25T19:50:12.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:50:12.965 INFO:tasks.cephadm:Failed to find podman. Using docker 2024-04-25T19:50:12.965 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker pull quay.io/samba.org/samba-ad-server:latest 2024-04-25T19:50:13.689 INFO:teuthology.orchestra.run.smithi150.stdout:latest: Pulling from samba.org/samba-ad-server 2024-04-25T19:50:13.689 INFO:teuthology.orchestra.run.smithi150.stdout:0aaf541ec2cb: Pulling fs layer 2024-04-25T19:50:13.689 INFO:teuthology.orchestra.run.smithi150.stdout:1de86a40d880: Pulling fs layer 2024-04-25T19:50:13.689 INFO:teuthology.orchestra.run.smithi150.stdout:20a2ae5038d4: Pulling fs layer 2024-04-25T19:50:13.689 INFO:teuthology.orchestra.run.smithi150.stdout:d7e8a3c0ddea: Pulling fs layer 2024-04-25T19:50:13.689 INFO:teuthology.orchestra.run.smithi150.stdout:16b55230dae6: Pulling fs layer 2024-04-25T19:50:13.689 INFO:teuthology.orchestra.run.smithi150.stdout:1f19703ab831: Pulling fs layer 2024-04-25T19:50:13.690 INFO:teuthology.orchestra.run.smithi150.stdout:d7e8a3c0ddea: Waiting 2024-04-25T19:50:13.690 INFO:teuthology.orchestra.run.smithi150.stdout:16b55230dae6: Waiting 2024-04-25T19:50:13.690 INFO:teuthology.orchestra.run.smithi150.stdout:1f19703ab831: Waiting 2024-04-25T19:50:13.853 INFO:teuthology.orchestra.run.smithi150.stdout:1de86a40d880: Download complete 2024-04-25T19:50:14.117 INFO:teuthology.orchestra.run.smithi150.stdout:d7e8a3c0ddea: Verifying Checksum 2024-04-25T19:50:14.117 INFO:teuthology.orchestra.run.smithi150.stdout:d7e8a3c0ddea: Download complete 2024-04-25T19:50:14.376 INFO:teuthology.orchestra.run.smithi150.stdout:16b55230dae6: Verifying Checksum 2024-04-25T19:50:14.376 INFO:teuthology.orchestra.run.smithi150.stdout:16b55230dae6: Download complete 2024-04-25T19:50:14.970 INFO:teuthology.orchestra.run.smithi150.stdout:20a2ae5038d4: Verifying Checksum 2024-04-25T19:50:14.970 INFO:teuthology.orchestra.run.smithi150.stdout:20a2ae5038d4: Download complete 2024-04-25T19:50:15.258 INFO:teuthology.orchestra.run.smithi150.stdout:1f19703ab831: Verifying Checksum 2024-04-25T19:50:15.258 INFO:teuthology.orchestra.run.smithi150.stdout:1f19703ab831: Download complete 2024-04-25T19:50:15.272 INFO:teuthology.orchestra.run.smithi150.stdout:0aaf541ec2cb: Verifying Checksum 2024-04-25T19:50:15.273 INFO:teuthology.orchestra.run.smithi150.stdout:0aaf541ec2cb: Download complete 2024-04-25T19:50:19.498 INFO:teuthology.orchestra.run.smithi150.stdout:0aaf541ec2cb: Pull complete 2024-04-25T19:50:19.649 INFO:teuthology.orchestra.run.smithi150.stdout:1de86a40d880: Pull complete 2024-04-25T19:50:23.770 INFO:teuthology.orchestra.run.smithi150.stdout:20a2ae5038d4: Pull complete 2024-04-25T19:50:23.930 INFO:teuthology.orchestra.run.smithi150.stdout:d7e8a3c0ddea: Pull complete 2024-04-25T19:50:24.116 INFO:teuthology.orchestra.run.smithi150.stdout:16b55230dae6: Pull complete 2024-04-25T19:50:26.073 INFO:teuthology.orchestra.run.smithi150.stdout:1f19703ab831: Pull complete 2024-04-25T19:50:26.186 INFO:teuthology.orchestra.run.smithi150.stdout:Digest: sha256:c7861a8444fc7e375fcc93a9fcc465fb8fa8db3f9be1e40c4b230877a91a9a4b 2024-04-25T19:50:26.247 INFO:teuthology.orchestra.run.smithi150.stdout:Status: Downloaded newer image for quay.io/samba.org/samba-ad-server:latest 2024-04-25T19:50:26.294 INFO:teuthology.orchestra.run.smithi150.stdout:quay.io/samba.org/samba-ad-server:latest 2024-04-25T19:50:26.297 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker pull quay.io/samba.org/samba-client:latest 2024-04-25T19:50:26.994 INFO:teuthology.orchestra.run.smithi150.stdout:latest: Pulling from samba.org/samba-client 2024-04-25T19:50:26.994 INFO:teuthology.orchestra.run.smithi150.stdout:0aaf541ec2cb: Already exists 2024-04-25T19:50:27.034 INFO:teuthology.orchestra.run.smithi150.stdout:eed49dabbc10: Pulling fs layer 2024-04-25T19:50:27.837 INFO:teuthology.orchestra.run.smithi150.stdout:eed49dabbc10: Download complete 2024-04-25T19:50:29.881 INFO:teuthology.orchestra.run.smithi150.stdout:eed49dabbc10: Pull complete 2024-04-25T19:50:30.010 INFO:teuthology.orchestra.run.smithi150.stdout:Digest: sha256:8d3a7d440b015d47a9adbd69f334f42b1b74da669ad0fc578502eed55c7b743b 2024-04-25T19:50:30.058 INFO:teuthology.orchestra.run.smithi150.stdout:Status: Downloaded newer image for quay.io/samba.org/samba-client:latest 2024-04-25T19:50:30.093 INFO:teuthology.orchestra.run.smithi150.stdout:quay.io/samba.org/samba-client:latest 2024-04-25T19:50:30.097 DEBUG:teuthology.orchestra.run.smithi150:> ss -lunH 2024-04-25T19:50:30.129 INFO:teuthology.orchestra.run.smithi150.stdout:UNCONN 0 0 0.0.0.0:46085 0.0.0.0:* 2024-04-25T19:50:30.129 INFO:teuthology.orchestra.run.smithi150.stdout:UNCONN 0 0 127.0.0.53%lo:53 0.0.0.0:* 2024-04-25T19:50:30.129 INFO:teuthology.orchestra.run.smithi150.stdout:UNCONN 0 0 172.21.15.150%enp3s0f1:68 0.0.0.0:* 2024-04-25T19:50:30.129 INFO:teuthology.orchestra.run.smithi150.stdout:UNCONN 0 0 0.0.0.0:68 0.0.0.0:* 2024-04-25T19:50:30.129 INFO:teuthology.orchestra.run.smithi150.stdout:UNCONN 0 0 0.0.0.0:111 0.0.0.0:* 2024-04-25T19:50:30.129 INFO:teuthology.orchestra.run.smithi150.stdout:UNCONN 0 0 172.21.15.150:123 0.0.0.0:* 2024-04-25T19:50:30.129 INFO:teuthology.orchestra.run.smithi150.stdout:UNCONN 0 0 127.0.0.1:123 0.0.0.0:* 2024-04-25T19:50:30.129 INFO:teuthology.orchestra.run.smithi150.stdout:UNCONN 0 0 0.0.0.0:123 0.0.0.0:* 2024-04-25T19:50:30.129 INFO:teuthology.orchestra.run.smithi150.stdout:UNCONN 0 0 127.0.0.1:705 0.0.0.0:* 2024-04-25T19:50:30.130 INFO:teuthology.orchestra.run.smithi150.stdout:UNCONN 0 0 0.0.0.0:39575 0.0.0.0:* 2024-04-25T19:50:30.130 INFO:teuthology.orchestra.run.smithi150.stdout:UNCONN 0 0 0.0.0.0:41685 0.0.0.0:* 2024-04-25T19:50:30.130 INFO:teuthology.orchestra.run.smithi150.stdout:UNCONN 0 0 0.0.0.0:43980 0.0.0.0:* 2024-04-25T19:50:30.130 INFO:teuthology.orchestra.run.smithi150.stdout:UNCONN 0 0 0.0.0.0:44447 0.0.0.0:* 2024-04-25T19:50:30.130 INFO:teuthology.orchestra.run.smithi150.stdout:UNCONN 0 0 [::]:45940 [::]:* 2024-04-25T19:50:30.130 INFO:teuthology.orchestra.run.smithi150.stdout:UNCONN 0 0 [::]:46611 [::]:* 2024-04-25T19:50:30.130 INFO:teuthology.orchestra.run.smithi150.stdout:UNCONN 0 0 [::]:111 [::]:* 2024-04-25T19:50:30.130 INFO:teuthology.orchestra.run.smithi150.stdout:UNCONN 0 0 [fe80::ec4:7aff:fe88:6e35]%enp3s0f1:123 [::]:* 2024-04-25T19:50:30.130 INFO:teuthology.orchestra.run.smithi150.stdout:UNCONN 0 0 [::1]:123 [::]:* 2024-04-25T19:50:30.130 INFO:teuthology.orchestra.run.smithi150.stdout:UNCONN 0 0 [::]:123 [::]:* 2024-04-25T19:50:30.130 INFO:teuthology.orchestra.run.smithi150.stdout:UNCONN 0 0 [::]:39018 [::]:* 2024-04-25T19:50:30.130 INFO:teuthology.orchestra.run.smithi150.stdout:UNCONN 0 0 [::]:39052 [::]:* 2024-04-25T19:50:30.130 INFO:teuthology.orchestra.run.smithi150.stdout:UNCONN 0 0 [::]:57468 [::]:* 2024-04-25T19:50:30.131 INFO:tasks.cephadm:Disabling systemd-resolved on smithi150 2024-04-25T19:50:30.131 DEBUG:teuthology.orchestra.run.smithi150:> sudo cat /etc/systemd/resolved.conf 2024-04-25T19:50:30.185 INFO:teuthology.orchestra.run.smithi150.stdout:# This file is part of systemd. 2024-04-25T19:50:30.185 INFO:teuthology.orchestra.run.smithi150.stdout:# 2024-04-25T19:50:30.185 INFO:teuthology.orchestra.run.smithi150.stdout:# systemd is free software; you can redistribute it and/or modify it under the 2024-04-25T19:50:30.185 INFO:teuthology.orchestra.run.smithi150.stdout:# terms of the GNU Lesser General Public License as published by the Free 2024-04-25T19:50:30.185 INFO:teuthology.orchestra.run.smithi150.stdout:# Software Foundation; either version 2.1 of the License, or (at your option) 2024-04-25T19:50:30.186 INFO:teuthology.orchestra.run.smithi150.stdout:# any later version. 2024-04-25T19:50:30.186 INFO:teuthology.orchestra.run.smithi150.stdout:# 2024-04-25T19:50:30.186 INFO:teuthology.orchestra.run.smithi150.stdout:# Entries in this file show the compile time defaults. Local configuration 2024-04-25T19:50:30.186 INFO:teuthology.orchestra.run.smithi150.stdout:# should be created by either modifying this file, or by creating "drop-ins" in 2024-04-25T19:50:30.186 INFO:teuthology.orchestra.run.smithi150.stdout:# the resolved.conf.d/ subdirectory. The latter is generally recommended. 2024-04-25T19:50:30.186 INFO:teuthology.orchestra.run.smithi150.stdout:# Defaults can be restored by simply deleting this file and all drop-ins. 2024-04-25T19:50:30.186 INFO:teuthology.orchestra.run.smithi150.stdout:# 2024-04-25T19:50:30.186 INFO:teuthology.orchestra.run.smithi150.stdout:# Use 'systemd-analyze cat-config systemd/resolved.conf' to display the full config. 2024-04-25T19:50:30.186 INFO:teuthology.orchestra.run.smithi150.stdout:# 2024-04-25T19:50:30.186 INFO:teuthology.orchestra.run.smithi150.stdout:# See resolved.conf(5) for details. 2024-04-25T19:50:30.187 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T19:50:30.187 INFO:teuthology.orchestra.run.smithi150.stdout:[Resolve] 2024-04-25T19:50:30.187 INFO:teuthology.orchestra.run.smithi150.stdout:# Some examples of DNS servers which may be used for DNS= and FallbackDNS=: 2024-04-25T19:50:30.187 INFO:teuthology.orchestra.run.smithi150.stdout:# Cloudflare: 1.1.1.1#cloudflare-dns.com 1.0.0.1#cloudflare-dns.com 2606:4700:4700::1111#cloudflare-dns.com 2606:4700:4700::1001#cloudflare-dns.com 2024-04-25T19:50:30.187 INFO:teuthology.orchestra.run.smithi150.stdout:# Google: 8.8.8.8#dns.google 8.8.4.4#dns.google 2001:4860:4860::8888#dns.google 2001:4860:4860::8844#dns.google 2024-04-25T19:50:30.187 INFO:teuthology.orchestra.run.smithi150.stdout:# Quad9: 9.9.9.9#dns.quad9.net 149.112.112.112#dns.quad9.net 2620:fe::fe#dns.quad9.net 2620:fe::9#dns.quad9.net 2024-04-25T19:50:30.187 INFO:teuthology.orchestra.run.smithi150.stdout:#DNS= 2024-04-25T19:50:30.187 INFO:teuthology.orchestra.run.smithi150.stdout:#FallbackDNS= 2024-04-25T19:50:30.187 INFO:teuthology.orchestra.run.smithi150.stdout:#Domains= 2024-04-25T19:50:30.187 INFO:teuthology.orchestra.run.smithi150.stdout:#DNSSEC=no 2024-04-25T19:50:30.187 INFO:teuthology.orchestra.run.smithi150.stdout:#DNSOverTLS=no 2024-04-25T19:50:30.187 INFO:teuthology.orchestra.run.smithi150.stdout:#MulticastDNS=no 2024-04-25T19:50:30.187 INFO:teuthology.orchestra.run.smithi150.stdout:#LLMNR=no 2024-04-25T19:50:30.187 INFO:teuthology.orchestra.run.smithi150.stdout:#Cache=no-negative 2024-04-25T19:50:30.188 INFO:teuthology.orchestra.run.smithi150.stdout:#CacheFromLocalhost=no 2024-04-25T19:50:30.188 INFO:teuthology.orchestra.run.smithi150.stdout:#DNSStubListener=yes 2024-04-25T19:50:30.188 INFO:teuthology.orchestra.run.smithi150.stdout:#DNSStubListenerExtra= 2024-04-25T19:50:30.188 INFO:teuthology.orchestra.run.smithi150.stdout:#ReadEtcHosts=yes 2024-04-25T19:50:30.188 INFO:teuthology.orchestra.run.smithi150.stdout:#ResolveUnicastSingleLabel=no 2024-04-25T19:50:30.189 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T19:50:30.189 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/systemd/resolved.conf 2024-04-25T19:50:30.259 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl restart systemd-resolved 2024-04-25T19:50:30.390 DEBUG:teuthology.orchestra.run.smithi150:> ss -lunH 2024-04-25T19:50:30.395 INFO:teuthology.orchestra.run.smithi150.stdout:UNCONN 0 0 0.0.0.0:46085 0.0.0.0:* 2024-04-25T19:50:30.395 INFO:teuthology.orchestra.run.smithi150.stdout:UNCONN 0 0 172.21.15.150%enp3s0f1:68 0.0.0.0:* 2024-04-25T19:50:30.396 INFO:teuthology.orchestra.run.smithi150.stdout:UNCONN 0 0 0.0.0.0:68 0.0.0.0:* 2024-04-25T19:50:30.396 INFO:teuthology.orchestra.run.smithi150.stdout:UNCONN 0 0 0.0.0.0:111 0.0.0.0:* 2024-04-25T19:50:30.396 INFO:teuthology.orchestra.run.smithi150.stdout:UNCONN 0 0 172.21.15.150:123 0.0.0.0:* 2024-04-25T19:50:30.396 INFO:teuthology.orchestra.run.smithi150.stdout:UNCONN 0 0 127.0.0.1:123 0.0.0.0:* 2024-04-25T19:50:30.396 INFO:teuthology.orchestra.run.smithi150.stdout:UNCONN 0 0 0.0.0.0:123 0.0.0.0:* 2024-04-25T19:50:30.396 INFO:teuthology.orchestra.run.smithi150.stdout:UNCONN 0 0 127.0.0.1:705 0.0.0.0:* 2024-04-25T19:50:30.396 INFO:teuthology.orchestra.run.smithi150.stdout:UNCONN 0 0 0.0.0.0:39575 0.0.0.0:* 2024-04-25T19:50:30.396 INFO:teuthology.orchestra.run.smithi150.stdout:UNCONN 0 0 0.0.0.0:41685 0.0.0.0:* 2024-04-25T19:50:30.396 INFO:teuthology.orchestra.run.smithi150.stdout:UNCONN 0 0 0.0.0.0:43980 0.0.0.0:* 2024-04-25T19:50:30.396 INFO:teuthology.orchestra.run.smithi150.stdout:UNCONN 0 0 0.0.0.0:44447 0.0.0.0:* 2024-04-25T19:50:30.396 INFO:teuthology.orchestra.run.smithi150.stdout:UNCONN 0 0 [::]:45940 [::]:* 2024-04-25T19:50:30.396 INFO:teuthology.orchestra.run.smithi150.stdout:UNCONN 0 0 [::]:46611 [::]:* 2024-04-25T19:50:30.396 INFO:teuthology.orchestra.run.smithi150.stdout:UNCONN 0 0 [::]:111 [::]:* 2024-04-25T19:50:30.396 INFO:teuthology.orchestra.run.smithi150.stdout:UNCONN 0 0 [fe80::ec4:7aff:fe88:6e35]%enp3s0f1:123 [::]:* 2024-04-25T19:50:30.397 INFO:teuthology.orchestra.run.smithi150.stdout:UNCONN 0 0 [::1]:123 [::]:* 2024-04-25T19:50:30.397 INFO:teuthology.orchestra.run.smithi150.stdout:UNCONN 0 0 [::]:123 [::]:* 2024-04-25T19:50:30.397 INFO:teuthology.orchestra.run.smithi150.stdout:UNCONN 0 0 [::]:39018 [::]:* 2024-04-25T19:50:30.397 INFO:teuthology.orchestra.run.smithi150.stdout:UNCONN 0 0 [::]:39052 [::]:* 2024-04-25T19:50:30.397 INFO:teuthology.orchestra.run.smithi150.stdout:UNCONN 0 0 [::]:57468 [::]:* 2024-04-25T19:50:30.397 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl restart docker 2024-04-25T19:50:30.456 INFO:teuthology.orchestra.run.smithi150.stderr:Warning: The unit file, source configuration file or drop-ins of docker.service changed on disk. Run 'systemctl daemon-reload' to reload units. 2024-04-25T19:50:36.124 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/samba/container/logs /var/lib/samba/container/data 2024-04-25T19:50:36.178 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/tmp/samba 2024-04-25T19:50:36.233 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T19:50:36.233 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/var/tmp/samba/container.json 2024-04-25T19:50:36.284 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker run -d --name=samba-ad --network=host --privileged --volume=/var/tmp/samba:/etc/samba-container:ro -eSAMBACC_CONFIG=/etc/samba-container/container.json quay.io/samba.org/samba-ad-server:latest 2024-04-25T19:50:36.850 INFO:teuthology.orchestra.run.smithi150.stdout:91683618f53fca2cdf52e9ae1d0e7408f584483ab17f8ebb85d77b923f96ce71 2024-04-25T19:50:37.730 INFO:tasks.cephadm:Probing SMB status of DC 172.21.15.150, idx=0 2024-04-25T19:50:37.730 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker run --rm --net=host --dns=172.21.15.150 -eKRB5_CONFIG=/dev/null quay.io/samba.org/samba-client:latest smbclient -U 'DOMAIN1\ckent%1115Rose.' //domain1.sink.test/sysvol -c ls 2024-04-25T19:50:38.950 INFO:teuthology.orchestra.run.smithi150.stderr:do_connect: Connection to domain1.sink.test failed (Error NT_STATUS_UNSUCCESSFUL) 2024-04-25T19:50:39.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:50:40.416 INFO:tasks.cephadm:Probing SMB status of DC 172.21.15.150, idx=1 2024-04-25T19:50:40.417 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker run --rm --net=host --dns=172.21.15.150 -eKRB5_CONFIG=/dev/null quay.io/samba.org/samba-client:latest smbclient -U 'DOMAIN1\ckent%1115Rose.' //domain1.sink.test/sysvol -c ls 2024-04-25T19:50:41.548 INFO:teuthology.orchestra.run.smithi150.stderr:do_connect: Connection to domain1.sink.test failed (Error NT_STATUS_UNSUCCESSFUL) 2024-04-25T19:50:42.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:50:43.524 INFO:tasks.cephadm:Probing SMB status of DC 172.21.15.150, idx=2 2024-04-25T19:50:43.524 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker run --rm --net=host --dns=172.21.15.150 -eKRB5_CONFIG=/dev/null quay.io/samba.org/samba-client:latest smbclient -U 'DOMAIN1\ckent%1115Rose.' //domain1.sink.test/sysvol -c ls 2024-04-25T19:50:44.517 INFO:teuthology.orchestra.run.smithi150.stderr:do_connect: Connection to domain1.sink.test failed (Error NT_STATUS_UNSUCCESSFUL) 2024-04-25T19:50:45.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:50:47.055 INFO:tasks.cephadm:Probing SMB status of DC 172.21.15.150, idx=3 2024-04-25T19:50:47.056 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker run --rm --net=host --dns=172.21.15.150 -eKRB5_CONFIG=/dev/null quay.io/samba.org/samba-client:latest smbclient -U 'DOMAIN1\ckent%1115Rose.' //domain1.sink.test/sysvol -c ls 2024-04-25T19:50:47.805 INFO:teuthology.orchestra.run.smithi150.stderr:do_connect: Connection to domain1.sink.test failed (Error NT_STATUS_UNSUCCESSFUL) 2024-04-25T19:50:48.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:50:52.470 INFO:tasks.cephadm:Probing SMB status of DC 172.21.15.150, idx=4 2024-04-25T19:50:52.470 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker run --rm --net=host --dns=172.21.15.150 -eKRB5_CONFIG=/dev/null quay.io/samba.org/samba-client:latest smbclient -U 'DOMAIN1\ckent%1115Rose.' //domain1.sink.test/sysvol -c ls 2024-04-25T19:50:53.657 INFO:teuthology.orchestra.run.smithi150.stderr:do_connect: Connection to domain1.sink.test failed (Error NT_STATUS_UNSUCCESSFUL) 2024-04-25T19:50:54.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:51:02.520 INFO:tasks.cephadm:Probing SMB status of DC 172.21.15.150, idx=5 2024-04-25T19:51:02.521 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker run --rm --net=host --dns=172.21.15.150 -eKRB5_CONFIG=/dev/null quay.io/samba.org/samba-client:latest smbclient -U 'DOMAIN1\ckent%1115Rose.' //domain1.sink.test/sysvol -c ls 2024-04-25T19:51:03.385 INFO:teuthology.orchestra.run.smithi150.stderr:do_connect: Connection to domain1.sink.test failed (Error NT_STATUS_UNSUCCESSFUL) 2024-04-25T19:51:04.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:51:20.260 INFO:tasks.cephadm:Probing SMB status of DC 172.21.15.150, idx=6 2024-04-25T19:51:20.261 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker run --rm --net=host --dns=172.21.15.150 -eKRB5_CONFIG=/dev/null quay.io/samba.org/samba-client:latest smbclient -U 'DOMAIN1\ckent%1115Rose.' //domain1.sink.test/sysvol -c ls 2024-04-25T19:51:21.675 INFO:teuthology.orchestra.run.smithi150.stdout: . D 0 Thu Apr 25 19:50:49 2024 2024-04-25T19:51:21.675 INFO:teuthology.orchestra.run.smithi150.stdout: .. D 0 Thu Apr 25 19:50:49 2024 2024-04-25T19:51:21.676 INFO:teuthology.orchestra.run.smithi150.stdout: domain1.sink.test D 0 Thu Apr 25 19:50:47 2024 2024-04-25T19:51:21.676 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T19:51:21.676 INFO:teuthology.orchestra.run.smithi150.stdout: 960311348 blocks of size 1024. 902342756 blocks available 2024-04-25T19:51:22.300 INFO:tasks.cephadm:SMB status probe succeeded 2024-04-25T19:51:22.300 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-25T19:51:22.310 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-only-match': ['CEPHADM_'], 'sha1': 'b48599b46c4ce29785962a8af5ddf30f6f38e57f'} 2024-04-25T19:51:22.310 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f 2024-04-25T19:51:22.311 INFO:tasks.cephadm:Cluster fsid is 3114c848-033d-11ef-bc93-c7b262605968 2024-04-25T19:51:22.311 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-25T19:51:22.311 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.132'} 2024-04-25T19:51:22.311 INFO:tasks.cephadm:First mon is mon.a on smithi132 2024-04-25T19:51:22.311 INFO:tasks.cephadm:First mgr is x 2024-04-25T19:51:22.311 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-25T19:51:22.311 DEBUG:teuthology.orchestra.run.smithi132:> sudo hostname $(hostname -s) 2024-04-25T19:51:22.327 DEBUG:teuthology.orchestra.run.smithi150:> sudo hostname $(hostname -s) 2024-04-25T19:51:22.342 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-25T19:51:22.342 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b48599b46c4ce29785962a8af5ddf30f6f38e57f 2024-04-25T19:51:22.564 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'b48599b46c4ce29785962a8af5ddf30f6f38e57f', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=jammy,DIST=jammy,MACHINE_SIZE=gigantic/78860/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3290-gb48599b4', 'node_name': '172.21.2.17+braggi17', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=jammy,DIST=jammy,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3290-gb48599b4-1jammy'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-24-1356/b48599b46c4ce29785962a8af5ddf30f6f38e57f/ubuntu/jammy/flavors/default/', 'distro_codename': 'jammy', 'modified': '2024-04-24 22:41:33.889439', 'distro_version': '22.04', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-phlogistonjohn-testing-2024-04-24-1356', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-phlogistonjohn-testing-2024-04-24-1356/b48599b46c4ce29785962a8af5ddf30f6f38e57f/ubuntu/jammy/flavors/default/', 'archs': ['x86_64'], 'distro': 'ubuntu'}] 2024-04-25T19:51:22.703 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-phlogistonjohn-testing-2024-04-24-1356, sha1 b48599b46c4ce29785962a8af5ddf30f6f38e57f from https://shaman.ceph.com/api/search/?project=ceph&distros=ubuntu%2F22.04%2Fx86_64&flavor=default&sha1=b48599b46c4ce29785962a8af5ddf30f6f38e57f 2024-04-25T19:51:22.705 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-phlogistonjohn-testing-2024-04-24-1356/b48599b46c4ce29785962a8af5ddf30f6f38e57f/ubuntu/jammy/x86_64/flavors/default/cephadm 2024-04-25T19:51:22.705 DEBUG:teuthology.orchestra.run.smithi132:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-phlogistonjohn-testing-2024-04-24-1356/b48599b46c4ce29785962a8af5ddf30f6f38e57f/ubuntu/jammy/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-25T19:51:23.742 INFO:teuthology.orchestra.run.smithi132.stdout:-rw-rw-r-- 1 ubuntu ubuntu 789108 Apr 25 19:51 /home/ubuntu/cephtest/cephadm 2024-04-25T19:51:23.742 DEBUG:teuthology.orchestra.run.smithi150:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-phlogistonjohn-testing-2024-04-24-1356/b48599b46c4ce29785962a8af5ddf30f6f38e57f/ubuntu/jammy/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-25T19:51:24.586 INFO:teuthology.orchestra.run.smithi150.stdout:-rw-rw-r-- 1 ubuntu ubuntu 789108 Apr 25 19:51 /home/ubuntu/cephtest/cephadm 2024-04-25T19:51:24.586 DEBUG:teuthology.orchestra.run.smithi132:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-25T19:51:24.594 DEBUG:teuthology.orchestra.run.smithi150:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-25T19:51:24.611 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f on all hosts... 2024-04-25T19:51:24.612 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f pull 2024-04-25T19:51:24.639 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f pull 2024-04-25T19:51:24.803 INFO:teuthology.orchestra.run.smithi132.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f... 2024-04-25T19:51:24.811 INFO:teuthology.orchestra.run.smithi150.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f... 2024-04-25T19:52:17.357 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-25T19:52:17.357 INFO:teuthology.orchestra.run.smithi132.stdout: "ceph_version": "ceph version 19.0.0-3290-gb48599b4 (b48599b46c4ce29785962a8af5ddf30f6f38e57f) squid (dev)", 2024-04-25T19:52:17.357 INFO:teuthology.orchestra.run.smithi132.stdout: "image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", 2024-04-25T19:52:17.357 INFO:teuthology.orchestra.run.smithi132.stdout: "repo_digests": [ 2024-04-25T19:52:17.357 INFO:teuthology.orchestra.run.smithi132.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a" 2024-04-25T19:52:17.357 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-04-25T19:52:17.358 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-25T19:52:25.260 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-04-25T19:52:25.260 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_version": "ceph version 19.0.0-3290-gb48599b4 (b48599b46c4ce29785962a8af5ddf30f6f38e57f) squid (dev)", 2024-04-25T19:52:25.260 INFO:teuthology.orchestra.run.smithi150.stdout: "image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", 2024-04-25T19:52:25.260 INFO:teuthology.orchestra.run.smithi150.stdout: "repo_digests": [ 2024-04-25T19:52:25.260 INFO:teuthology.orchestra.run.smithi150.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a" 2024-04-25T19:52:25.260 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-04-25T19:52:25.260 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-04-25T19:52:25.295 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /etc/ceph 2024-04-25T19:52:25.310 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /etc/ceph 2024-04-25T19:52:25.323 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 777 /etc/ceph 2024-04-25T19:52:25.364 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 777 /etc/ceph 2024-04-25T19:52:25.376 INFO:tasks.cephadm:Writing seed config... 2024-04-25T19:52:25.377 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-25T19:52:25.377 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-25T19:52:25.377 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-25T19:52:25.377 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-25T19:52:25.377 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-25T19:52:25.377 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-25T19:52:25.377 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-25T19:52:25.378 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-25T19:52:25.378 DEBUG:teuthology.orchestra.run.smithi132:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-25T19:52:25.413 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 3114c848-033d-11ef-bc93-c7b262605968 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-04-25T19:52:25.414 DEBUG:teuthology.orchestra.run.smithi132:mon.a> sudo journalctl -f -n 0 -u ceph-3114c848-033d-11ef-bc93-c7b262605968@mon.a.service 2024-04-25T19:52:25.458 DEBUG:teuthology.orchestra.run.smithi132:mgr.x> sudo journalctl -f -n 0 -u ceph-3114c848-033d-11ef-bc93-c7b262605968@mgr.x.service 2024-04-25T19:52:25.502 INFO:tasks.cephadm:Bootstrapping... 2024-04-25T19:52:25.502 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f -v bootstrap --fsid 3114c848-033d-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.132 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-25T19:52:25.711 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-04-25T19:52:25.711 INFO:teuthology.orchestra.run.smithi132.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f', '-v', 'bootstrap', '--fsid', '3114c848-033d-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.132', '--skip-admin-label'] 2024-04-25T19:52:25.712 INFO:teuthology.orchestra.run.smithi132.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-25T19:52:25.712 INFO:teuthology.orchestra.run.smithi132.stdout:Verifying podman|docker is present... 2024-04-25T19:52:25.712 INFO:teuthology.orchestra.run.smithi132.stdout:Verifying lvm2 is present... 2024-04-25T19:52:25.713 INFO:teuthology.orchestra.run.smithi132.stdout:Verifying time synchronization is in place... 2024-04-25T19:52:25.717 INFO:teuthology.orchestra.run.smithi132.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-25T19:52:25.717 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-25T19:52:25.721 INFO:teuthology.orchestra.run.smithi132.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-25T19:52:25.721 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stdout inactive 2024-04-25T19:52:25.725 INFO:teuthology.orchestra.run.smithi132.stdout:Non-zero exit code 1 from systemctl is-enabled chronyd.service 2024-04-25T19:52:25.725 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stderr Failed to get unit file state for chronyd.service: No such file or directory 2024-04-25T19:52:25.729 INFO:teuthology.orchestra.run.smithi132.stdout:Non-zero exit code 3 from systemctl is-active chronyd.service 2024-04-25T19:52:25.729 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stdout inactive 2024-04-25T19:52:25.734 INFO:teuthology.orchestra.run.smithi132.stdout:Non-zero exit code 1 from systemctl is-enabled systemd-timesyncd.service 2024-04-25T19:52:25.734 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stdout masked 2024-04-25T19:52:25.737 INFO:teuthology.orchestra.run.smithi132.stdout:Non-zero exit code 3 from systemctl is-active systemd-timesyncd.service 2024-04-25T19:52:25.738 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stdout inactive 2024-04-25T19:52:25.741 INFO:teuthology.orchestra.run.smithi132.stdout:Non-zero exit code 1 from systemctl is-enabled ntpd.service 2024-04-25T19:52:25.741 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stderr Failed to get unit file state for ntpd.service: No such file or directory 2024-04-25T19:52:25.744 INFO:teuthology.orchestra.run.smithi132.stdout:Non-zero exit code 3 from systemctl is-active ntpd.service 2024-04-25T19:52:25.745 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stdout inactive 2024-04-25T19:52:25.748 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stdout enabled 2024-04-25T19:52:25.752 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stdout active 2024-04-25T19:52:25.752 INFO:teuthology.orchestra.run.smithi132.stdout:Unit ntp.service is enabled and running 2024-04-25T19:52:25.752 INFO:teuthology.orchestra.run.smithi132.stdout:Repeating the final host check... 2024-04-25T19:52:25.752 INFO:teuthology.orchestra.run.smithi132.stdout:docker (/usr/bin/docker) is present 2024-04-25T19:52:25.752 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl is present 2024-04-25T19:52:25.753 INFO:teuthology.orchestra.run.smithi132.stdout:lvcreate is present 2024-04-25T19:52:25.755 INFO:teuthology.orchestra.run.smithi132.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-25T19:52:25.755 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-25T19:52:25.760 INFO:teuthology.orchestra.run.smithi132.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-25T19:52:25.760 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stdout inactive 2024-04-25T19:52:25.764 INFO:teuthology.orchestra.run.smithi132.stdout:Non-zero exit code 1 from systemctl is-enabled chronyd.service 2024-04-25T19:52:25.764 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stderr Failed to get unit file state for chronyd.service: No such file or directory 2024-04-25T19:52:25.769 INFO:teuthology.orchestra.run.smithi132.stdout:Non-zero exit code 3 from systemctl is-active chronyd.service 2024-04-25T19:52:25.769 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stdout inactive 2024-04-25T19:52:25.772 INFO:teuthology.orchestra.run.smithi132.stdout:Non-zero exit code 1 from systemctl is-enabled systemd-timesyncd.service 2024-04-25T19:52:25.772 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stdout masked 2024-04-25T19:52:25.777 INFO:teuthology.orchestra.run.smithi132.stdout:Non-zero exit code 3 from systemctl is-active systemd-timesyncd.service 2024-04-25T19:52:25.777 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stdout inactive 2024-04-25T19:52:25.780 INFO:teuthology.orchestra.run.smithi132.stdout:Non-zero exit code 1 from systemctl is-enabled ntpd.service 2024-04-25T19:52:25.780 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stderr Failed to get unit file state for ntpd.service: No such file or directory 2024-04-25T19:52:25.785 INFO:teuthology.orchestra.run.smithi132.stdout:Non-zero exit code 3 from systemctl is-active ntpd.service 2024-04-25T19:52:25.785 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stdout inactive 2024-04-25T19:52:25.789 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stdout enabled 2024-04-25T19:52:25.793 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stdout active 2024-04-25T19:52:25.793 INFO:teuthology.orchestra.run.smithi132.stdout:Unit ntp.service is enabled and running 2024-04-25T19:52:25.794 INFO:teuthology.orchestra.run.smithi132.stdout:Host looks OK 2024-04-25T19:52:25.794 INFO:teuthology.orchestra.run.smithi132.stdout:Cluster fsid: 3114c848-033d-11ef-bc93-c7b262605968 2024-04-25T19:52:25.794 INFO:teuthology.orchestra.run.smithi132.stdout:Acquiring lock 140424794499360 on /run/cephadm/3114c848-033d-11ef-bc93-c7b262605968.lock 2024-04-25T19:52:25.794 INFO:teuthology.orchestra.run.smithi132.stdout:Lock 140424794499360 acquired on /run/cephadm/3114c848-033d-11ef-bc93-c7b262605968.lock 2024-04-25T19:52:25.794 INFO:teuthology.orchestra.run.smithi132.stdout:Verifying IP 172.21.15.132 port 3300 ... 2024-04-25T19:52:25.794 INFO:teuthology.orchestra.run.smithi132.stdout:Verifying IP 172.21.15.132 port 6789 ... 2024-04-25T19:52:25.794 INFO:teuthology.orchestra.run.smithi132.stdout:Base mon IP(s) is [172.21.15.132:3300, 172.21.15.132:6789], mon addrv is [v2:172.21.15.132:3300,v1:172.21.15.132:6789] 2024-04-25T19:52:25.796 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 2024-04-25T19:52:25.796 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.132 metric 100 2024-04-25T19:52:25.796 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/sbin/ip: stdout 158.69.67.47 via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.132 metric 100 2024-04-25T19:52:25.796 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/sbin/ip: stdout 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 2024-04-25T19:52:25.797 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.132 metric 100 2024-04-25T19:52:25.797 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/sbin/ip: stdout 172.21.0.1 dev enp3s0f1 proto dhcp scope link src 172.21.15.132 metric 100 2024-04-25T19:52:25.797 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/sbin/ip: stdout 172.21.0.2 dev enp3s0f1 proto dhcp scope link src 172.21.15.132 metric 100 2024-04-25T19:52:25.797 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/sbin/ip: stdout 172.21.15.254 dev enp3s0f1 proto dhcp scope link src 172.21.15.132 metric 100 2024-04-25T19:52:25.798 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-25T19:52:25.798 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-04-25T19:52:25.798 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 expires 1781sec pref medium 2024-04-25T19:52:25.800 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-25T19:52:25.800 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-25T19:52:25.801 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-25T19:52:25.801 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-25T19:52:25.801 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/sbin/ip: stdout inet6 fe80::ec4:7aff:fe88:773f/64 scope link 2024-04-25T19:52:25.801 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-25T19:52:25.802 INFO:teuthology.orchestra.run.smithi132.stdout:Mon IP `172.21.15.132` is in CIDR network `172.21.0.0/20` 2024-04-25T19:52:25.802 INFO:teuthology.orchestra.run.smithi132.stdout:Mon IP `172.21.15.132` is in CIDR network `172.21.0.0/20` 2024-04-25T19:52:25.802 INFO:teuthology.orchestra.run.smithi132.stdout:Mon IP `172.21.15.132` is in CIDR network `172.21.0.1/32` 2024-04-25T19:52:25.802 INFO:teuthology.orchestra.run.smithi132.stdout:Mon IP `172.21.15.132` is in CIDR network `172.21.0.1/32` 2024-04-25T19:52:25.802 INFO:teuthology.orchestra.run.smithi132.stdout:Mon IP `172.21.15.132` is in CIDR network `172.21.0.2/32` 2024-04-25T19:52:25.802 INFO:teuthology.orchestra.run.smithi132.stdout:Mon IP `172.21.15.132` is in CIDR network `172.21.0.2/32` 2024-04-25T19:52:25.802 INFO:teuthology.orchestra.run.smithi132.stdout:Mon IP `172.21.15.132` is in CIDR network `172.21.15.254/32` 2024-04-25T19:52:25.803 INFO:teuthology.orchestra.run.smithi132.stdout:Mon IP `172.21.15.132` is in CIDR network `172.21.15.254/32` 2024-04-25T19:52:25.803 INFO:teuthology.orchestra.run.smithi132.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20', '172.21.0.1/32', '172.21.0.1/32', '172.21.0.2/32', '172.21.0.2/32', '172.21.15.254/32', '172.21.15.254/32'] 2024-04-25T19:52:25.803 INFO:teuthology.orchestra.run.smithi132.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-25T19:52:25.803 INFO:teuthology.orchestra.run.smithi132.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f... 2024-04-25T19:52:26.085 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/docker: stdout b48599b46c4ce29785962a8af5ddf30f6f38e57f: Pulling from ceph-ci/ceph 2024-04-25T19:52:26.085 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/docker: stdout Digest: sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a 2024-04-25T19:52:26.085 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/docker: stdout Status: Image is up to date for quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f 2024-04-25T19:52:26.085 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/docker: stdout quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f 2024-04-25T19:52:27.559 INFO:teuthology.orchestra.run.smithi132.stdout:ceph: stdout ceph version 19.0.0-3290-gb48599b4 (b48599b46c4ce29785962a8af5ddf30f6f38e57f) squid (dev) 2024-04-25T19:52:27.560 INFO:teuthology.orchestra.run.smithi132.stdout:Ceph version: ceph version 19.0.0-3290-gb48599b4 (b48599b46c4ce29785962a8af5ddf30f6f38e57f) squid (dev) 2024-04-25T19:52:27.560 INFO:teuthology.orchestra.run.smithi132.stdout:Extracting ceph user uid/gid from container image... 2024-04-25T19:52:28.771 INFO:teuthology.orchestra.run.smithi132.stdout:stat: stdout 167 167 2024-04-25T19:52:28.771 INFO:teuthology.orchestra.run.smithi132.stdout:Creating initial keys... 2024-04-25T19:52:30.042 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-authtool: stdout AQB9tCpmUdvhHBAAn1gGGudU/CT3JFiX4Q/yVw== 2024-04-25T19:52:31.281 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-authtool: stdout AQB+tCpmom7FLRAA2GP8IANvN/oXhtOQTtgF/A== 2024-04-25T19:52:32.626 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-authtool: stdout AQCAtCpmvATZAxAAlKDuPmcy09zT1XvAmTn1LA== 2024-04-25T19:52:32.627 INFO:teuthology.orchestra.run.smithi132.stdout:Creating initial monmap... 2024-04-25T19:52:33.873 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-25T19:52:33.873 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-25T19:52:33.873 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 3114c848-033d-11ef-bc93-c7b262605968 2024-04-25T19:52:33.873 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-25T19:52:33.873 INFO:teuthology.orchestra.run.smithi132.stdout:monmaptool for a [v2:172.21.15.132:3300,v1:172.21.15.132:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-25T19:52:33.873 INFO:teuthology.orchestra.run.smithi132.stdout:setting min_mon_release = quincy 2024-04-25T19:52:33.873 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/monmaptool: set fsid to 3114c848-033d-11ef-bc93-c7b262605968 2024-04-25T19:52:33.873 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-25T19:52:33.874 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T19:52:33.874 INFO:teuthology.orchestra.run.smithi132.stdout:Creating mon... 2024-04-25T19:52:35.121 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.582+0000 7f39f4f3bc80 0 set uid:gid to 167:167 (ceph:ceph) 2024-04-25T19:52:35.121 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.582+0000 7f39f4f3bc80 1 imported monmap: 2024-04-25T19:52:35.121 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr epoch 0 2024-04-25T19:52:35.121 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr fsid 3114c848-033d-11ef-bc93-c7b262605968 2024-04-25T19:52:35.121 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr last_changed 2024-04-25T19:52:33.352311+0000 2024-04-25T19:52:35.122 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr created 2024-04-25T19:52:33.352311+0000 2024-04-25T19:52:35.122 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr min_mon_release 17 (quincy) 2024-04-25T19:52:35.122 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr election_strategy: 1 2024-04-25T19:52:35.122 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr 0: [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] mon.a 2024-04-25T19:52:35.122 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr 2024-04-25T19:52:35.122 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.582+0000 7f39f4f3bc80 0 /usr/bin/ceph-mon: set fsid to 3114c848-033d-11ef-bc93-c7b262605968 2024-04-25T19:52:35.122 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: RocksDB version: 7.9.2 2024-04-25T19:52:35.122 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr 2024-04-25T19:52:35.137 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Git sha 0 2024-04-25T19:52:35.137 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Compile date 2024-04-24 20:01:12 2024-04-25T19:52:35.137 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: DB SUMMARY 2024-04-25T19:52:35.137 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr 2024-04-25T19:52:35.137 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: DB Session ID: 47BVHUKGOKBRPC7BU2W4 2024-04-25T19:52:35.137 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr 2024-04-25T19:52:35.137 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 0, files: 2024-04-25T19:52:35.137 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr 2024-04-25T19:52:35.138 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 2024-04-25T19:52:35.138 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr 2024-04-25T19:52:35.138 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.error_if_exists: 0 2024-04-25T19:52:35.138 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.create_if_missing: 1 2024-04-25T19:52:35.138 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.paranoid_checks: 1 2024-04-25T19:52:35.138 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.flush_verify_memtable_count: 1 2024-04-25T19:52:35.138 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-25T19:52:35.138 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-25T19:52:35.138 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.env: 0x56430f7c2cc0 2024-04-25T19:52:35.138 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.fs: PosixFileSystem 2024-04-25T19:52:35.138 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.info_log: 0x5643100eb020 2024-04-25T19:52:35.138 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.max_file_opening_threads: 16 2024-04-25T19:52:35.138 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.statistics: (nil) 2024-04-25T19:52:35.139 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.use_fsync: 0 2024-04-25T19:52:35.139 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.max_log_file_size: 0 2024-04-25T19:52:35.139 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-25T19:52:35.139 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.log_file_time_to_roll: 0 2024-04-25T19:52:35.139 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.keep_log_file_num: 1000 2024-04-25T19:52:35.139 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.recycle_log_file_num: 0 2024-04-25T19:52:35.139 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.allow_fallocate: 1 2024-04-25T19:52:35.139 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.allow_mmap_reads: 0 2024-04-25T19:52:35.139 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.allow_mmap_writes: 0 2024-04-25T19:52:35.139 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.use_direct_reads: 0 2024-04-25T19:52:35.139 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-25T19:52:35.139 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.create_missing_column_families: 0 2024-04-25T19:52:35.139 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.db_log_dir: 2024-04-25T19:52:35.140 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.wal_dir: 2024-04-25T19:52:35.140 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.table_cache_numshardbits: 6 2024-04-25T19:52:35.140 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-04-25T19:52:35.140 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-04-25T19:52:35.140 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-25T19:52:35.140 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-25T19:52:35.140 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-04-25T19:52:35.140 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.advise_random_on_open: 1 2024-04-25T19:52:35.140 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.db_write_buffer_size: 0 2024-04-25T19:52:35.140 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.write_buffer_manager: 0x5643100baaa0 2024-04-25T19:52:35.140 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-25T19:52:35.140 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-25T19:52:35.140 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.use_adaptive_mutex: 0 2024-04-25T19:52:35.140 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.rate_limiter: (nil) 2024-04-25T19:52:35.141 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-25T19:52:35.141 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.wal_recovery_mode: 2 2024-04-25T19:52:35.141 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.enable_thread_tracking: 0 2024-04-25T19:52:35.141 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.enable_pipelined_write: 0 2024-04-25T19:52:35.141 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.unordered_write: 0 2024-04-25T19:52:35.141 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-25T19:52:35.141 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-25T19:52:35.141 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-25T19:52:35.141 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-25T19:52:35.141 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.row_cache: None 2024-04-25T19:52:35.141 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.wal_filter: None 2024-04-25T19:52:35.141 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-25T19:52:35.141 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.allow_ingest_behind: 0 2024-04-25T19:52:35.141 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.two_write_queues: 0 2024-04-25T19:52:35.142 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.manual_wal_flush: 0 2024-04-25T19:52:35.142 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.wal_compression: 0 2024-04-25T19:52:35.142 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.atomic_flush: 0 2024-04-25T19:52:35.142 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-25T19:52:35.142 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.persist_stats_to_disk: 0 2024-04-25T19:52:35.142 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-04-25T19:52:35.142 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.log_readahead_size: 0 2024-04-25T19:52:35.142 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-25T19:52:35.142 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.best_efforts_recovery: 0 2024-04-25T19:52:35.142 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-25T19:52:35.142 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-25T19:52:35.142 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.allow_data_in_errors: 0 2024-04-25T19:52:35.142 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.db_host_id: __hostname__ 2024-04-25T19:52:35.142 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.enforce_single_del_contracts: true 2024-04-25T19:52:35.143 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.max_background_jobs: 2 2024-04-25T19:52:35.143 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.max_background_compactions: -1 2024-04-25T19:52:35.143 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.max_subcompactions: 1 2024-04-25T19:52:35.143 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-25T19:52:35.143 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-25T19:52:35.143 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.delayed_write_rate : 16777216 2024-04-25T19:52:35.143 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.max_total_wal_size: 0 2024-04-25T19:52:35.143 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-25T19:52:35.143 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.stats_dump_period_sec: 600 2024-04-25T19:52:35.143 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.stats_persist_period_sec: 600 2024-04-25T19:52:35.143 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-25T19:52:35.143 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.max_open_files: -1 2024-04-25T19:52:35.143 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.bytes_per_sync: 0 2024-04-25T19:52:35.143 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-04-25T19:52:35.144 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-04-25T19:52:35.144 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.compaction_readahead_size: 0 2024-04-25T19:52:35.144 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Options.max_background_flushes: -1 2024-04-25T19:52:35.144 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Compression algorithms supported: 2024-04-25T19:52:35.144 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-25T19:52:35.144 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: kZSTD supported: 0 2024-04-25T19:52:35.144 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: kXpressCompression supported: 0 2024-04-25T19:52:35.144 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: kLZ4HCCompression supported: 1 2024-04-25T19:52:35.144 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: kZlibCompression supported: 1 2024-04-25T19:52:35.144 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: kSnappyCompression supported: 1 2024-04-25T19:52:35.144 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: kLZ4Compression supported: 1 2024-04-25T19:52:35.144 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: kBZip2Compression supported: 0 2024-04-25T19:52:35.144 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-04-25T19:52:35.144 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: DMutex implementation: pthread_mutex_t 2024-04-25T19:52:35.145 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.586+0000 7f39f4f3bc80 4 rocksdb: [db/db_impl/db_impl_open.cc:317] Creating manifest 1 2024-04-25T19:52:35.145 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr 2024-04-25T19:52:35.145 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 2024-04-25T19:52:35.145 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr 2024-04-25T19:52:35.145 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-25T19:52:35.145 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr 2024-04-25T19:52:35.145 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-25T19:52:35.145 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.merge_operator: 2024-04-25T19:52:35.145 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.compaction_filter: None 2024-04-25T19:52:35.145 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.compaction_filter_factory: None 2024-04-25T19:52:35.145 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.sst_partitioner_factory: None 2024-04-25T19:52:35.145 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-04-25T19:52:35.145 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.table_factory: BlockBasedTable 2024-04-25T19:52:35.145 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5643100eb1e0) 2024-04-25T19:52:35.145 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr cache_index_and_filter_blocks: 1 2024-04-25T19:52:35.146 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr cache_index_and_filter_blocks_with_high_priority: 0 2024-04-25T19:52:35.146 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-25T19:52:35.146 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr pin_top_level_index_and_filter: 1 2024-04-25T19:52:35.146 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr index_type: 0 2024-04-25T19:52:35.146 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr data_block_index_type: 0 2024-04-25T19:52:35.146 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr index_shortening: 1 2024-04-25T19:52:35.146 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr data_block_hash_table_util_ratio: 0.750000 2024-04-25T19:52:35.146 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr checksum: 4 2024-04-25T19:52:35.146 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr no_block_cache: 0 2024-04-25T19:52:35.146 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr block_cache: 0x5643100ed1f0 2024-04-25T19:52:35.146 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr block_cache_name: BinnedLRUCache 2024-04-25T19:52:35.146 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr block_cache_options: 2024-04-25T19:52:35.146 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr capacity : 536870912 2024-04-25T19:52:35.146 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr num_shard_bits : 4 2024-04-25T19:52:35.147 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr strict_capacity_limit : 0 2024-04-25T19:52:35.147 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr high_pri_pool_ratio: 0.000 2024-04-25T19:52:35.147 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr block_cache_compressed: (nil) 2024-04-25T19:52:35.147 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr persistent_cache: (nil) 2024-04-25T19:52:35.147 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr block_size: 4096 2024-04-25T19:52:35.147 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr block_size_deviation: 10 2024-04-25T19:52:35.147 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr block_restart_interval: 16 2024-04-25T19:52:35.147 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr index_block_restart_interval: 1 2024-04-25T19:52:35.147 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr metadata_block_size: 4096 2024-04-25T19:52:35.147 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr partition_filters: 0 2024-04-25T19:52:35.147 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr use_delta_encoding: 1 2024-04-25T19:52:35.147 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr filter_policy: bloomfilter 2024-04-25T19:52:35.147 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr whole_key_filtering: 1 2024-04-25T19:52:35.147 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr verify_compression: 0 2024-04-25T19:52:35.147 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr read_amp_bytes_per_bit: 0 2024-04-25T19:52:35.148 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr format_version: 5 2024-04-25T19:52:35.148 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr enable_index_compression: 1 2024-04-25T19:52:35.148 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr block_align: 0 2024-04-25T19:52:35.148 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr max_auto_readahead_size: 262144 2024-04-25T19:52:35.148 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr prepopulate_block_cache: 0 2024-04-25T19:52:35.148 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr initial_auto_readahead_size: 8192 2024-04-25T19:52:35.160 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr num_file_reads_for_auto_readahead: 2 2024-04-25T19:52:35.160 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr 2024-04-25T19:52:35.160 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.write_buffer_size: 33554432 2024-04-25T19:52:35.160 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.max_write_buffer_number: 2 2024-04-25T19:52:35.160 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.compression: NoCompression 2024-04-25T19:52:35.161 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.bottommost_compression: Disabled 2024-04-25T19:52:35.161 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.prefix_extractor: nullptr 2024-04-25T19:52:35.161 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-25T19:52:35.161 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.num_levels: 7 2024-04-25T19:52:35.161 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-25T19:52:35.161 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-25T19:52:35.161 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-25T19:52:35.161 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-25T19:52:35.161 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-25T19:52:35.161 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-25T19:52:35.161 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-25T19:52:35.161 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-25T19:52:35.161 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-25T19:52:35.161 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-25T19:52:35.162 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-25T19:52:35.162 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-25T19:52:35.162 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.compression_opts.window_bits: -14 2024-04-25T19:52:35.162 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.compression_opts.level: 32767 2024-04-25T19:52:35.162 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.compression_opts.strategy: 0 2024-04-25T19:52:35.162 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-25T19:52:35.162 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-25T19:52:35.162 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-25T19:52:35.162 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-25T19:52:35.162 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.compression_opts.enabled: false 2024-04-25T19:52:35.162 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-25T19:52:35.162 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-25T19:52:35.162 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-25T19:52:35.162 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-25T19:52:35.163 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.target_file_size_base: 67108864 2024-04-25T19:52:35.163 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.target_file_size_multiplier: 1 2024-04-25T19:52:35.163 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-25T19:52:35.163 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-25T19:52:35.163 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-25T19:52:35.163 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-25T19:52:35.163 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-25T19:52:35.163 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-25T19:52:35.163 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-25T19:52:35.163 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-25T19:52:35.163 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-25T19:52:35.163 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-25T19:52:35.163 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-25T19:52:35.164 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-25T19:52:35.164 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-25T19:52:35.164 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.arena_block_size: 1048576 2024-04-25T19:52:35.164 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-25T19:52:35.164 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-25T19:52:35.164 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.disable_auto_compactions: 0 2024-04-25T19:52:35.164 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-25T19:52:35.164 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-25T19:52:35.164 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-25T19:52:35.164 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-25T19:52:35.164 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-25T19:52:35.165 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-25T19:52:35.165 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-25T19:52:35.165 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-25T19:52:35.165 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-25T19:52:35.165 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-25T19:52:35.165 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-25T19:52:35.165 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.inplace_update_support: 0 2024-04-25T19:52:35.165 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-04-25T19:52:35.165 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-25T19:52:35.165 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-25T19:52:35.165 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.memtable_huge_page_size: 0 2024-04-25T19:52:35.165 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.bloom_locality: 0 2024-04-25T19:52:35.165 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.max_successive_merges: 0 2024-04-25T19:52:35.165 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-04-25T19:52:35.166 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.paranoid_file_checks: 0 2024-04-25T19:52:35.166 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.force_consistency_checks: 1 2024-04-25T19:52:35.166 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.report_bg_io_stats: 0 2024-04-25T19:52:35.166 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.ttl: 2592000 2024-04-25T19:52:35.166 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-04-25T19:52:35.166 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-25T19:52:35.166 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-25T19:52:35.166 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.enable_blob_files: false 2024-04-25T19:52:35.166 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.min_blob_size: 0 2024-04-25T19:52:35.166 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.blob_file_size: 268435456 2024-04-25T19:52:35.166 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.blob_compression_type: NoCompression 2024-04-25T19:52:35.166 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.enable_blob_garbage_collection: false 2024-04-25T19:52:35.166 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-25T19:52:35.166 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-25T19:52:35.167 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-25T19:52:35.167 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.blob_file_starting_level: 0 2024-04-25T19:52:35.167 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-25T19:52:35.167 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-25T19:52:35.167 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr 2024-04-25T19:52:35.167 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-25T19:52:35.167 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr 2024-04-25T19:52:35.167 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: fa2dfb67-d911-46c7-9073-033b53264d1a 2024-04-25T19:52:35.167 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr 2024-04-25T19:52:35.167 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: [db/version_set.cc:5047] Creating manifest 5 2024-04-25T19:52:35.167 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr 2024-04-25T19:52:35.167 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5643101e4000 2024-04-25T19:52:35.167 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39f4f3bc80 4 rocksdb: DB pointer 0x5643101d4000 2024-04-25T19:52:35.167 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39e509b700 4 rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-25T19:52:35.168 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.590+0000 7f39e509b700 4 rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-25T19:52:35.168 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr ** DB Stats ** 2024-04-25T19:52:35.168 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T19:52:35.168 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-25T19:52:35.168 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T19:52:35.168 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T19:52:35.168 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-25T19:52:35.168 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T19:52:35.168 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T19:52:35.168 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr 2024-04-25T19:52:35.168 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr ** Compaction Stats [default] ** 2024-04-25T19:52:35.168 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T19:52:35.168 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-25T19:52:35.168 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-25T19:52:35.169 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-25T19:52:35.169 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr 2024-04-25T19:52:35.169 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr ** Compaction Stats [default] ** 2024-04-25T19:52:35.169 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T19:52:35.169 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-25T19:52:35.169 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr 2024-04-25T19:52:35.169 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-25T19:52:35.169 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr 2024-04-25T19:52:35.169 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T19:52:35.169 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr Flush(GB): cumulative 0.000, interval 0.000 2024-04-25T19:52:35.169 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr AddFile(GB): cumulative 0.000, interval 0.000 2024-04-25T19:52:35.169 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr AddFile(Total Files): cumulative 0, interval 0 2024-04-25T19:52:35.169 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr AddFile(L0 Files): cumulative 0, interval 0 2024-04-25T19:52:35.169 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr AddFile(Keys): cumulative 0, interval 0 2024-04-25T19:52:35.169 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T19:52:35.170 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T19:52:35.170 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-25T19:52:35.170 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr Block cache BinnedLRUCache@0x5643100ed1f0#7 capacity: 512.00 MB usage: 0.00 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.3e-05 secs_since: 0 2024-04-25T19:52:35.170 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr Block cache entry stats(count,size,portion): Misc(1,0.00 KB,0%) 2024-04-25T19:52:35.170 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr 2024-04-25T19:52:35.170 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr ** File Read Latency Histogram By Level [default] ** 2024-04-25T19:52:35.170 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr 2024-04-25T19:52:35.170 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.594+0000 7f39f4f3bc80 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2024-04-25T19:52:35.170 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.594+0000 7f39f4f3bc80 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2024-04-25T19:52:35.170 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-25T19:52:34.594+0000 7f39f4f3bc80 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-a for mon.a 2024-04-25T19:52:35.170 INFO:teuthology.orchestra.run.smithi132.stdout:create mon.a on 2024-04-25T19:52:35.606 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /etc/systemd/system/ceph.target. 2024-04-25T19:52:35.844 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-3114c848-033d-11ef-bc93-c7b262605968.target -> /etc/systemd/system/ceph-3114c848-033d-11ef-bc93-c7b262605968.target. 2024-04-25T19:52:35.844 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-3114c848-033d-11ef-bc93-c7b262605968.target -> /etc/systemd/system/ceph-3114c848-033d-11ef-bc93-c7b262605968.target. 2024-04-25T19:52:36.136 INFO:teuthology.orchestra.run.smithi132.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-3114c848-033d-11ef-bc93-c7b262605968@mon.a 2024-04-25T19:52:36.136 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stderr Failed to reset failed state of unit ceph-3114c848-033d-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-3114c848-033d-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-25T19:52:36.334 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-3114c848-033d-11ef-bc93-c7b262605968.target.wants/ceph-3114c848-033d-11ef-bc93-c7b262605968@mon.a.service -> /etc/systemd/system/ceph-3114c848-033d-11ef-bc93-c7b262605968@.service. 2024-04-25T19:52:36.343 INFO:teuthology.orchestra.run.smithi132.stdout:firewalld does not appear to be present 2024-04-25T19:52:36.343 INFO:teuthology.orchestra.run.smithi132.stdout:Not possible to enable service . firewalld.service is not available 2024-04-25T19:52:36.343 INFO:teuthology.orchestra.run.smithi132.stdout:Waiting for mon to start... 2024-04-25T19:52:36.343 INFO:teuthology.orchestra.run.smithi132.stdout:Waiting for mon... 2024-04-25T19:52:37.747 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:37 smithi132 bash[18119]: cluster 2024-04-25T19:52:37.537115+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-04-25T19:52:38.280 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout cluster: 2024-04-25T19:52:38.280 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout id: 3114c848-033d-11ef-bc93-c7b262605968 2024-04-25T19:52:38.280 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-25T19:52:38.280 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout 2024-04-25T19:52:38.280 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout services: 2024-04-25T19:52:38.280 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.18102s) 2024-04-25T19:52:38.280 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-25T19:52:38.280 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-25T19:52:38.280 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout 2024-04-25T19:52:38.280 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout data: 2024-04-25T19:52:38.281 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-25T19:52:38.281 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-25T19:52:38.281 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-25T19:52:38.281 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout pgs: 2024-04-25T19:52:38.282 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout 2024-04-25T19:52:38.282 INFO:teuthology.orchestra.run.smithi132.stdout:mon is available 2024-04-25T19:52:38.282 INFO:teuthology.orchestra.run.smithi132.stdout:Assimilating anything we can from ceph.conf... 2024-04-25T19:52:38.841 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:38 smithi132 bash[18119]: cluster 2024-04-25T19:52:37.557099+0000 mon.a (mon.0) 2 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-04-25T19:52:38.841 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:38 smithi132 bash[18119]: cluster 2024-04-25T19:52:37.557099+0000 mon.a (mon.0) 2 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-04-25T19:52:38.841 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:38 smithi132 bash[18119]: cluster 2024-04-25T19:52:37.557328+0000 mon.a (mon.0) 3 : cluster [DBG] monmap epoch 1 2024-04-25T19:52:38.841 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:38 smithi132 bash[18119]: cluster 2024-04-25T19:52:37.557328+0000 mon.a (mon.0) 3 : cluster [DBG] monmap epoch 1 2024-04-25T19:52:38.841 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:38 smithi132 bash[18119]: cluster 2024-04-25T19:52:37.557341+0000 mon.a (mon.0) 4 : cluster [DBG] fsid 3114c848-033d-11ef-bc93-c7b262605968 2024-04-25T19:52:38.841 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:38 smithi132 bash[18119]: cluster 2024-04-25T19:52:37.557341+0000 mon.a (mon.0) 4 : cluster [DBG] fsid 3114c848-033d-11ef-bc93-c7b262605968 2024-04-25T19:52:38.841 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:38 smithi132 bash[18119]: cluster 2024-04-25T19:52:37.557349+0000 mon.a (mon.0) 5 : cluster [DBG] last_changed 2024-04-25T19:52:33.352311+0000 2024-04-25T19:52:38.841 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:38 smithi132 bash[18119]: cluster 2024-04-25T19:52:37.557349+0000 mon.a (mon.0) 5 : cluster [DBG] last_changed 2024-04-25T19:52:33.352311+0000 2024-04-25T19:52:38.841 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:38 smithi132 bash[18119]: cluster 2024-04-25T19:52:37.557357+0000 mon.a (mon.0) 6 : cluster [DBG] created 2024-04-25T19:52:33.352311+0000 2024-04-25T19:52:38.841 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:38 smithi132 bash[18119]: cluster 2024-04-25T19:52:37.557357+0000 mon.a (mon.0) 6 : cluster [DBG] created 2024-04-25T19:52:33.352311+0000 2024-04-25T19:52:38.842 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:38 smithi132 bash[18119]: cluster 2024-04-25T19:52:37.557366+0000 mon.a (mon.0) 7 : cluster [DBG] min_mon_release 19 (squid) 2024-04-25T19:52:38.842 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:38 smithi132 bash[18119]: cluster 2024-04-25T19:52:37.557366+0000 mon.a (mon.0) 7 : cluster [DBG] min_mon_release 19 (squid) 2024-04-25T19:52:38.842 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:38 smithi132 bash[18119]: cluster 2024-04-25T19:52:37.557373+0000 mon.a (mon.0) 8 : cluster [DBG] election_strategy: 1 2024-04-25T19:52:38.842 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:38 smithi132 bash[18119]: cluster 2024-04-25T19:52:37.557373+0000 mon.a (mon.0) 8 : cluster [DBG] election_strategy: 1 2024-04-25T19:52:38.842 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:38 smithi132 bash[18119]: cluster 2024-04-25T19:52:37.557382+0000 mon.a (mon.0) 9 : cluster [DBG] 0: [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] mon.a 2024-04-25T19:52:38.842 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:38 smithi132 bash[18119]: cluster 2024-04-25T19:52:37.557382+0000 mon.a (mon.0) 9 : cluster [DBG] 0: [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] mon.a 2024-04-25T19:52:38.842 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:38 smithi132 bash[18119]: cluster 2024-04-25T19:52:37.559798+0000 mon.a (mon.0) 10 : cluster [DBG] fsmap 2024-04-25T19:52:38.842 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:38 smithi132 bash[18119]: cluster 2024-04-25T19:52:37.559798+0000 mon.a (mon.0) 10 : cluster [DBG] fsmap 2024-04-25T19:52:38.842 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:38 smithi132 bash[18119]: cluster 2024-04-25T19:52:37.562341+0000 mon.a (mon.0) 11 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-04-25T19:52:38.842 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:38 smithi132 bash[18119]: cluster 2024-04-25T19:52:37.562341+0000 mon.a (mon.0) 11 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-04-25T19:52:38.842 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:38 smithi132 bash[18119]: cluster 2024-04-25T19:52:37.562839+0000 mon.a (mon.0) 12 : cluster [DBG] mgrmap e1: no daemons active 2024-04-25T19:52:38.842 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:38 smithi132 bash[18119]: cluster 2024-04-25T19:52:37.562839+0000 mon.a (mon.0) 12 : cluster [DBG] mgrmap e1: no daemons active 2024-04-25T19:52:38.842 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:38 smithi132 bash[18119]: audit 2024-04-25T19:52:37.738066+0000 mon.a (mon.0) 13 : audit [DBG] from='client.? 172.21.15.132:0/3336665197' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-25T19:52:38.843 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:38 smithi132 bash[18119]: audit 2024-04-25T19:52:37.738066+0000 mon.a (mon.0) 13 : audit [DBG] from='client.? 172.21.15.132:0/3336665197' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-25T19:52:39.761 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout 2024-04-25T19:52:39.761 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout [global] 2024-04-25T19:52:39.761 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout fsid = 3114c848-033d-11ef-bc93-c7b262605968 2024-04-25T19:52:39.761 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-25T19:52:39.761 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.132:3300,v1:172.21.15.132:6789] 2024-04-25T19:52:39.762 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-25T19:52:39.762 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-25T19:52:39.762 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-25T19:52:39.762 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-25T19:52:39.762 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout 2024-04-25T19:52:39.762 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-25T19:52:39.763 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-25T19:52:39.763 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout 2024-04-25T19:52:39.763 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout [osd] 2024-04-25T19:52:39.763 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-25T19:52:39.763 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-25T19:52:39.763 INFO:teuthology.orchestra.run.smithi132.stdout:Generating new minimal ceph.conf... 2024-04-25T19:52:40.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:39 smithi132 bash[18119]: audit 2024-04-25T19:52:39.168354+0000 mon.a (mon.0) 14 : audit [INF] from='client.? 172.21.15.132:0/3497160013' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-25T19:52:40.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:39 smithi132 bash[18119]: audit 2024-04-25T19:52:39.168354+0000 mon.a (mon.0) 14 : audit [INF] from='client.? 172.21.15.132:0/3497160013' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-25T19:52:40.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:39 smithi132 bash[18119]: audit 2024-04-25T19:52:39.174103+0000 mon.a (mon.0) 15 : audit [INF] from='client.? 172.21.15.132:0/3497160013' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-25T19:52:40.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:39 smithi132 bash[18119]: audit 2024-04-25T19:52:39.174103+0000 mon.a (mon.0) 15 : audit [INF] from='client.? 172.21.15.132:0/3497160013' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-25T19:52:41.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:40 smithi132 bash[18119]: audit 2024-04-25T19:52:40.799038+0000 mon.a (mon.0) 16 : audit [DBG] from='client.? 172.21.15.132:0/3095160220' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:52:41.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:40 smithi132 bash[18119]: audit 2024-04-25T19:52:40.799038+0000 mon.a (mon.0) 16 : audit [DBG] from='client.? 172.21.15.132:0/3095160220' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:52:41.348 INFO:teuthology.orchestra.run.smithi132.stdout:Restarting the monitor... 2024-04-25T19:52:41.628 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:41 smithi132 systemd[1]: Stopping Ceph mon.a for 3114c848-033d-11ef-bc93-c7b262605968... 2024-04-25T19:52:41.629 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:41 smithi132 bash[18119]: debug 2024-04-25T19:52:41.453+0000 7f3b67982700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-04-25T19:52:41.629 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:41 smithi132 bash[18119]: debug 2024-04-25T19:52:41.453+0000 7f3b67982700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-25T19:52:42.203 INFO:teuthology.orchestra.run.smithi132.stdout:Setting public_network to 172.21.0.1/32,172.21.15.254/32,172.21.0.0/20,172.21.0.2/32 in mon config section 2024-04-25T19:52:42.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:42 smithi132 bash[18503]: ceph-3114c848-033d-11ef-bc93-c7b262605968-mon-a 2024-04-25T19:52:42.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:42 smithi132 bash[18566]: ceph-3114c848-033d-11ef-bc93-c7b262605968-mon-a 2024-04-25T19:52:42.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:42 smithi132 systemd[1]: ceph-3114c848-033d-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-25T19:52:42.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:42 smithi132 systemd[1]: Stopped Ceph mon.a for 3114c848-033d-11ef-bc93-c7b262605968. 2024-04-25T19:52:42.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:42 smithi132 systemd[1]: Started Ceph mon.a for 3114c848-033d-11ef-bc93-c7b262605968. 2024-04-25T19:52:43.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.265+0000 7fdd0cf21c80 0 set uid:gid to 167:167 (ceph:ceph) 2024-04-25T19:52:43.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.265+0000 7fdd0cf21c80 0 ceph version 19.0.0-3290-gb48599b4 (b48599b46c4ce29785962a8af5ddf30f6f38e57f) squid (dev), process ceph-mon, pid 7 2024-04-25T19:52:43.522 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.265+0000 7fdd0cf21c80 0 pidfile_write: ignore empty --pid-file 2024-04-25T19:52:43.522 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 0 load: jerasure load: lrc 2024-04-25T19:52:43.522 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: RocksDB version: 7.9.2 2024-04-25T19:52:43.522 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Git sha 0 2024-04-25T19:52:43.522 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Compile date 2024-04-24 20:01:12 2024-04-25T19:52:43.522 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: DB SUMMARY 2024-04-25T19:52:43.522 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: DB Session ID: RNK16FAOGA0DKW5LCJN4 2024-04-25T19:52:43.522 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: CURRENT file: CURRENT 2024-04-25T19:52:43.522 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: IDENTITY file: IDENTITY 2024-04-25T19:52:43.522 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-25T19:52:43.522 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-25T19:52:43.522 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 88035 ; 2024-04-25T19:52:43.523 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.error_if_exists: 0 2024-04-25T19:52:43.523 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.create_if_missing: 0 2024-04-25T19:52:43.523 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.paranoid_checks: 1 2024-04-25T19:52:43.523 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.flush_verify_memtable_count: 1 2024-04-25T19:52:43.523 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-25T19:52:43.523 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-25T19:52:43.523 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.env: 0x5609c3c59cc0 2024-04-25T19:52:43.523 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.fs: PosixFileSystem 2024-04-25T19:52:43.523 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.info_log: 0x5609c488ddc0 2024-04-25T19:52:43.523 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.max_file_opening_threads: 16 2024-04-25T19:52:43.523 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.statistics: (nil) 2024-04-25T19:52:43.523 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.use_fsync: 0 2024-04-25T19:52:43.523 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.max_log_file_size: 0 2024-04-25T19:52:43.524 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-25T19:52:43.524 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.log_file_time_to_roll: 0 2024-04-25T19:52:43.524 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.keep_log_file_num: 1000 2024-04-25T19:52:43.524 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.recycle_log_file_num: 0 2024-04-25T19:52:43.524 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.allow_fallocate: 1 2024-04-25T19:52:43.524 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.allow_mmap_reads: 0 2024-04-25T19:52:43.525 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.allow_mmap_writes: 0 2024-04-25T19:52:43.525 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.use_direct_reads: 0 2024-04-25T19:52:43.525 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-25T19:52:43.525 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.create_missing_column_families: 0 2024-04-25T19:52:43.525 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.db_log_dir: 2024-04-25T19:52:43.525 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.wal_dir: 2024-04-25T19:52:43.525 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.table_cache_numshardbits: 6 2024-04-25T19:52:43.525 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-04-25T19:52:43.525 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-04-25T19:52:43.525 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-25T19:52:43.525 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-25T19:52:43.525 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-04-25T19:52:43.525 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.advise_random_on_open: 1 2024-04-25T19:52:43.525 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.db_write_buffer_size: 0 2024-04-25T19:52:43.526 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.write_buffer_manager: 0x5609c485f0e0 2024-04-25T19:52:43.526 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-25T19:52:43.526 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-25T19:52:43.526 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.use_adaptive_mutex: 0 2024-04-25T19:52:43.526 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.rate_limiter: (nil) 2024-04-25T19:52:43.526 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-25T19:52:43.526 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.wal_recovery_mode: 2 2024-04-25T19:52:43.526 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.enable_thread_tracking: 0 2024-04-25T19:52:43.526 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.enable_pipelined_write: 0 2024-04-25T19:52:43.526 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.unordered_write: 0 2024-04-25T19:52:43.526 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-25T19:52:43.526 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-25T19:52:43.526 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-25T19:52:43.527 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-25T19:52:43.527 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.row_cache: None 2024-04-25T19:52:43.527 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.wal_filter: None 2024-04-25T19:52:43.527 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-25T19:52:43.527 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.allow_ingest_behind: 0 2024-04-25T19:52:43.527 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.two_write_queues: 0 2024-04-25T19:52:43.527 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.manual_wal_flush: 0 2024-04-25T19:52:43.527 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.wal_compression: 0 2024-04-25T19:52:43.527 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.atomic_flush: 0 2024-04-25T19:52:43.527 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-25T19:52:43.527 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.persist_stats_to_disk: 0 2024-04-25T19:52:43.527 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-04-25T19:52:43.527 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.log_readahead_size: 0 2024-04-25T19:52:43.527 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-25T19:52:43.527 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.best_efforts_recovery: 0 2024-04-25T19:52:43.528 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-25T19:52:43.528 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-25T19:52:43.528 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.allow_data_in_errors: 0 2024-04-25T19:52:43.528 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.db_host_id: __hostname__ 2024-04-25T19:52:43.528 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.enforce_single_del_contracts: true 2024-04-25T19:52:43.528 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.max_background_jobs: 2 2024-04-25T19:52:43.528 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.max_background_compactions: -1 2024-04-25T19:52:43.528 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.max_subcompactions: 1 2024-04-25T19:52:43.528 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-25T19:52:43.528 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-25T19:52:43.528 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.delayed_write_rate : 16777216 2024-04-25T19:52:43.528 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.max_total_wal_size: 0 2024-04-25T19:52:43.528 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-25T19:52:43.529 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.stats_dump_period_sec: 600 2024-04-25T19:52:43.529 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.stats_persist_period_sec: 600 2024-04-25T19:52:43.529 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-25T19:52:43.529 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.max_open_files: -1 2024-04-25T19:52:43.529 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.bytes_per_sync: 0 2024-04-25T19:52:43.529 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-04-25T19:52:43.529 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-04-25T19:52:43.529 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.compaction_readahead_size: 0 2024-04-25T19:52:43.529 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.max_background_flushes: -1 2024-04-25T19:52:43.529 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Compression algorithms supported: 2024-04-25T19:52:43.529 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-25T19:52:43.530 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: kZSTD supported: 0 2024-04-25T19:52:43.530 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: kXpressCompression supported: 0 2024-04-25T19:52:43.530 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: kLZ4HCCompression supported: 1 2024-04-25T19:52:43.530 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: kZlibCompression supported: 1 2024-04-25T19:52:43.530 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: kSnappyCompression supported: 1 2024-04-25T19:52:43.530 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: kLZ4Compression supported: 1 2024-04-25T19:52:43.530 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: kBZip2Compression supported: 0 2024-04-25T19:52:43.530 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-04-25T19:52:43.530 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: DMutex implementation: pthread_mutex_t 2024-04-25T19:52:43.530 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-25T19:52:43.531 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-25T19:52:43.532 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-25T19:52:43.532 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.merge_operator: 2024-04-25T19:52:43.532 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.compaction_filter: None 2024-04-25T19:52:43.532 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.compaction_filter_factory: None 2024-04-25T19:52:43.532 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.sst_partitioner_factory: None 2024-04-25T19:52:43.532 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-04-25T19:52:43.532 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.table_factory: BlockBasedTable 2024-04-25T19:52:43.532 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5609c488df80) 2024-04-25T19:52:43.532 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: cache_index_and_filter_blocks: 1 2024-04-25T19:52:43.532 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-25T19:52:43.532 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-25T19:52:43.532 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: pin_top_level_index_and_filter: 1 2024-04-25T19:52:43.532 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: index_type: 0 2024-04-25T19:52:43.533 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: data_block_index_type: 0 2024-04-25T19:52:43.533 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: index_shortening: 1 2024-04-25T19:52:43.533 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: data_block_hash_table_util_ratio: 0.750000 2024-04-25T19:52:43.533 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: checksum: 4 2024-04-25T19:52:43.533 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: no_block_cache: 0 2024-04-25T19:52:43.533 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: block_cache: 0x5609c488f1f0 2024-04-25T19:52:43.533 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: block_cache_name: BinnedLRUCache 2024-04-25T19:52:43.533 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: block_cache_options: 2024-04-25T19:52:43.533 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: capacity : 536870912 2024-04-25T19:52:43.533 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: num_shard_bits : 4 2024-04-25T19:52:43.533 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: strict_capacity_limit : 0 2024-04-25T19:52:43.533 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: high_pri_pool_ratio: 0.000 2024-04-25T19:52:43.533 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: block_cache_compressed: (nil) 2024-04-25T19:52:43.533 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: persistent_cache: (nil) 2024-04-25T19:52:43.533 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: block_size: 4096 2024-04-25T19:52:43.534 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: block_size_deviation: 10 2024-04-25T19:52:43.534 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: block_restart_interval: 16 2024-04-25T19:52:43.534 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: index_block_restart_interval: 1 2024-04-25T19:52:43.534 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: metadata_block_size: 4096 2024-04-25T19:52:43.534 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: partition_filters: 0 2024-04-25T19:52:43.534 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: use_delta_encoding: 1 2024-04-25T19:52:43.534 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: filter_policy: bloomfilter 2024-04-25T19:52:43.534 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: whole_key_filtering: 1 2024-04-25T19:52:43.534 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: verify_compression: 0 2024-04-25T19:52:43.534 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: read_amp_bytes_per_bit: 0 2024-04-25T19:52:43.534 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: format_version: 5 2024-04-25T19:52:43.535 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: enable_index_compression: 1 2024-04-25T19:52:43.535 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: block_align: 0 2024-04-25T19:52:43.535 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: max_auto_readahead_size: 262144 2024-04-25T19:52:43.535 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: prepopulate_block_cache: 0 2024-04-25T19:52:43.535 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: initial_auto_readahead_size: 8192 2024-04-25T19:52:43.535 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: num_file_reads_for_auto_readahead: 2 2024-04-25T19:52:43.535 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.write_buffer_size: 33554432 2024-04-25T19:52:43.535 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.max_write_buffer_number: 2 2024-04-25T19:52:43.535 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.compression: NoCompression 2024-04-25T19:52:43.535 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.bottommost_compression: Disabled 2024-04-25T19:52:43.535 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.prefix_extractor: nullptr 2024-04-25T19:52:43.535 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-25T19:52:43.535 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.num_levels: 7 2024-04-25T19:52:43.535 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-25T19:52:43.536 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-25T19:52:43.536 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-25T19:52:43.536 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-25T19:52:43.536 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-25T19:52:43.536 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-25T19:52:43.536 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-25T19:52:43.536 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-25T19:52:43.536 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-25T19:52:43.536 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-25T19:52:43.536 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-25T19:52:43.536 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-25T19:52:43.536 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.compression_opts.window_bits: -14 2024-04-25T19:52:43.536 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.compression_opts.level: 32767 2024-04-25T19:52:43.536 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.compression_opts.strategy: 0 2024-04-25T19:52:43.536 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-25T19:52:43.537 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-25T19:52:43.537 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-25T19:52:43.537 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-25T19:52:43.537 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.compression_opts.enabled: false 2024-04-25T19:52:43.537 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-25T19:52:43.537 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-25T19:52:43.537 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-25T19:52:43.537 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-25T19:52:43.537 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.target_file_size_base: 67108864 2024-04-25T19:52:43.537 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.target_file_size_multiplier: 1 2024-04-25T19:52:43.537 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-25T19:52:43.537 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-25T19:52:43.537 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-25T19:52:43.537 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-25T19:52:43.538 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-25T19:52:43.538 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-25T19:52:43.538 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-25T19:52:43.538 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-25T19:52:43.538 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-25T19:52:43.538 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-25T19:52:43.538 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-25T19:52:43.538 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-25T19:52:43.538 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-25T19:52:43.538 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.arena_block_size: 1048576 2024-04-25T19:52:43.538 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-25T19:52:43.538 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-25T19:52:43.538 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.disable_auto_compactions: 0 2024-04-25T19:52:43.538 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-25T19:52:43.539 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-25T19:52:43.539 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-25T19:52:43.539 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-25T19:52:43.539 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-25T19:52:43.539 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-25T19:52:43.539 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-25T19:52:43.539 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-25T19:52:43.539 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-25T19:52:43.539 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-25T19:52:43.539 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-25T19:52:43.539 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.inplace_update_support: 0 2024-04-25T19:52:43.539 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-04-25T19:52:43.539 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-25T19:52:43.539 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-25T19:52:43.539 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.memtable_huge_page_size: 0 2024-04-25T19:52:43.540 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.bloom_locality: 0 2024-04-25T19:52:43.540 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.max_successive_merges: 0 2024-04-25T19:52:43.540 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-04-25T19:52:43.540 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.paranoid_file_checks: 0 2024-04-25T19:52:43.540 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.force_consistency_checks: 1 2024-04-25T19:52:43.540 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.report_bg_io_stats: 0 2024-04-25T19:52:43.540 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.ttl: 2592000 2024-04-25T19:52:43.540 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-04-25T19:52:43.540 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-25T19:52:43.540 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-25T19:52:43.540 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.enable_blob_files: false 2024-04-25T19:52:43.541 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.min_blob_size: 0 2024-04-25T19:52:43.541 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.blob_file_size: 268435456 2024-04-25T19:52:43.541 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.blob_compression_type: NoCompression 2024-04-25T19:52:43.541 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.enable_blob_garbage_collection: false 2024-04-25T19:52:43.541 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-25T19:52:43.541 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-25T19:52:43.541 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-25T19:52:43.541 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.blob_file_starting_level: 0 2024-04-25T19:52:43.541 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-25T19:52:43.541 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-25T19:52:43.541 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-25T19:52:43.541 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.269+0000 7fdd0cf21c80 4 rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: fa2dfb67-d911-46c7-9073-033b53264d1a 2024-04-25T19:52:43.541 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.273+0000 7fdd0cf21c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1714074763276714, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-25T19:52:43.542 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.273+0000 7fdd0cf21c80 4 rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-25T19:52:43.542 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.273+0000 7fdd0cf21c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1714074763277840, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 84805, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 254, "table_properties": {"data_size": 82945, "index_size": 244, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 9921, "raw_average_key_size": 46, "raw_value_size": 77266, "raw_average_value_size": 362, "num_data_blocks": 11, "num_entries": 213, "num_filter_entries": 213, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714074763, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "fa2dfb67-d911-46c7-9073-033b53264d1a", "db_session_id": "RNK16FAOGA0DKW5LCJN4", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-25T19:52:43.542 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.273+0000 7fdd0cf21c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1714074763277951, "job": 1, "event": "recovery_finished"} 2024-04-25T19:52:43.542 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.273+0000 7fdd0cf21c80 4 rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-25T19:52:43.542 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.273+0000 7fdd0cf21c80 4 rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-25T19:52:43.542 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.273+0000 7fdd0cf21c80 4 rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5609c49b0000 2024-04-25T19:52:43.542 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.273+0000 7fdd0cf21c80 4 rocksdb: DB pointer 0x5609c499c000 2024-04-25T19:52:43.542 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.273+0000 7fdcfb679700 4 rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-25T19:52:43.542 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.273+0000 7fdcfb679700 4 rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-25T19:52:43.542 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: ** DB Stats ** 2024-04-25T19:52:43.542 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T19:52:43.542 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-25T19:52:43.542 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T19:52:43.542 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T19:52:43.543 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-25T19:52:43.543 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T19:52:43.543 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T19:52:43.543 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: ** Compaction Stats [default] ** 2024-04-25T19:52:43.543 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T19:52:43.543 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-25T19:52:43.543 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: L0 2/0 84.68 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 127.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T19:52:43.543 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: Sum 2/0 84.68 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 127.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T19:52:43.543 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 127.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T19:52:43.543 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: ** Compaction Stats [default] ** 2024-04-25T19:52:43.543 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T19:52:43.543 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-25T19:52:43.543 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 127.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T19:52:43.543 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-25T19:52:43.544 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T19:52:43.544 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: Flush(GB): cumulative 0.000, interval 0.000 2024-04-25T19:52:43.544 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-25T19:52:43.544 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: AddFile(Total Files): cumulative 0, interval 0 2024-04-25T19:52:43.544 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: AddFile(L0 Files): cumulative 0, interval 0 2024-04-25T19:52:43.544 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: AddFile(Keys): cumulative 0, interval 0 2024-04-25T19:52:43.544 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: Cumulative compaction: 0.00 GB write, 19.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T19:52:43.544 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: Interval compaction: 0.00 GB write, 19.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T19:52:43.544 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-25T19:52:43.544 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: Block cache BinnedLRUCache@0x5609c488f1f0#7 capacity: 512.00 MB usage: 1.20 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.4e-05 secs_since: 0 2024-04-25T19:52:43.544 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.44 KB,8.34465e-05%) Misc(1,0.00 KB,0%) 2024-04-25T19:52:43.544 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: ** File Read Latency Histogram By Level [default] ** 2024-04-25T19:52:43.544 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.273+0000 7fdd0cf21c80 0 starting mon.a rank 0 at public addrs [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] at bind addrs [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 3114c848-033d-11ef-bc93-c7b262605968 2024-04-25T19:52:43.544 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.277+0000 7fdd0cf21c80 1 mon.a@-1(???) e1 preinit fsid 3114c848-033d-11ef-bc93-c7b262605968 2024-04-25T19:52:43.545 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.277+0000 7fdd0cf21c80 0 mon.a@-1(???).mds e1 new map 2024-04-25T19:52:43.545 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.277+0000 7fdd0cf21c80 0 mon.a@-1(???).mds e1 print_map 2024-04-25T19:52:43.545 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: e1 2024-04-25T19:52:43.545 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T19:52:43.545 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T19:52:43.545 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: legacy client fscid: -1 2024-04-25T19:52:43.545 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: 2024-04-25T19:52:43.545 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: No filesystems configured 2024-04-25T19:52:43.545 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.277+0000 7fdd0cf21c80 0 mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-25T19:52:43.545 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.277+0000 7fdd0cf21c80 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T19:52:43.545 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.277+0000 7fdd0cf21c80 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T19:52:43.545 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.277+0000 7fdd0cf21c80 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T19:52:43.545 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: debug 2024-04-25T19:52:43.277+0000 7fdd0cf21c80 1 mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-25T19:52:43.545 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: cluster 2024-04-25T19:52:43.305235+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-04-25T19:52:43.546 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: cluster 2024-04-25T19:52:43.305235+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-04-25T19:52:43.546 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: cluster 2024-04-25T19:52:43.305346+0000 mon.a (mon.0) 2 : cluster [DBG] monmap epoch 1 2024-04-25T19:52:43.546 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: cluster 2024-04-25T19:52:43.305346+0000 mon.a (mon.0) 2 : cluster [DBG] monmap epoch 1 2024-04-25T19:52:43.546 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: cluster 2024-04-25T19:52:43.305357+0000 mon.a (mon.0) 3 : cluster [DBG] fsid 3114c848-033d-11ef-bc93-c7b262605968 2024-04-25T19:52:43.546 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: cluster 2024-04-25T19:52:43.305357+0000 mon.a (mon.0) 3 : cluster [DBG] fsid 3114c848-033d-11ef-bc93-c7b262605968 2024-04-25T19:52:43.546 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: cluster 2024-04-25T19:52:43.305374+0000 mon.a (mon.0) 4 : cluster [DBG] last_changed 2024-04-25T19:52:33.352311+0000 2024-04-25T19:52:43.546 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: cluster 2024-04-25T19:52:43.305374+0000 mon.a (mon.0) 4 : cluster [DBG] last_changed 2024-04-25T19:52:33.352311+0000 2024-04-25T19:52:43.546 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: cluster 2024-04-25T19:52:43.305388+0000 mon.a (mon.0) 5 : cluster [DBG] created 2024-04-25T19:52:33.352311+0000 2024-04-25T19:52:43.546 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: cluster 2024-04-25T19:52:43.305388+0000 mon.a (mon.0) 5 : cluster [DBG] created 2024-04-25T19:52:33.352311+0000 2024-04-25T19:52:43.546 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: cluster 2024-04-25T19:52:43.305400+0000 mon.a (mon.0) 6 : cluster [DBG] min_mon_release 19 (squid) 2024-04-25T19:52:43.546 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: cluster 2024-04-25T19:52:43.305400+0000 mon.a (mon.0) 6 : cluster [DBG] min_mon_release 19 (squid) 2024-04-25T19:52:43.546 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: cluster 2024-04-25T19:52:43.305413+0000 mon.a (mon.0) 7 : cluster [DBG] election_strategy: 1 2024-04-25T19:52:43.546 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: cluster 2024-04-25T19:52:43.305413+0000 mon.a (mon.0) 7 : cluster [DBG] election_strategy: 1 2024-04-25T19:52:43.547 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: cluster 2024-04-25T19:52:43.305427+0000 mon.a (mon.0) 8 : cluster [DBG] 0: [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] mon.a 2024-04-25T19:52:43.547 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: cluster 2024-04-25T19:52:43.305427+0000 mon.a (mon.0) 8 : cluster [DBG] 0: [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] mon.a 2024-04-25T19:52:43.547 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: cluster 2024-04-25T19:52:43.325118+0000 mon.a (mon.0) 9 : cluster [DBG] fsmap 2024-04-25T19:52:43.547 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: cluster 2024-04-25T19:52:43.325118+0000 mon.a (mon.0) 9 : cluster [DBG] fsmap 2024-04-25T19:52:43.547 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: cluster 2024-04-25T19:52:43.325171+0000 mon.a (mon.0) 10 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-04-25T19:52:43.547 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: cluster 2024-04-25T19:52:43.325171+0000 mon.a (mon.0) 10 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-04-25T19:52:43.547 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: cluster 2024-04-25T19:52:43.325899+0000 mon.a (mon.0) 11 : cluster [DBG] mgrmap e1: no daemons active 2024-04-25T19:52:43.547 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:43 smithi132 bash[18607]: cluster 2024-04-25T19:52:43.325899+0000 mon.a (mon.0) 11 : cluster [DBG] mgrmap e1: no daemons active 2024-04-25T19:52:44.233 INFO:teuthology.orchestra.run.smithi132.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-25T19:52:44.259 INFO:teuthology.orchestra.run.smithi132.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-25T19:52:44.259 INFO:teuthology.orchestra.run.smithi132.stdout:Creating mgr... 2024-04-25T19:52:44.259 INFO:teuthology.orchestra.run.smithi132.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-25T19:52:44.260 INFO:teuthology.orchestra.run.smithi132.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-25T19:52:44.573 INFO:teuthology.orchestra.run.smithi132.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-3114c848-033d-11ef-bc93-c7b262605968@mgr.x 2024-04-25T19:52:44.573 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stderr Failed to reset failed state of unit ceph-3114c848-033d-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-3114c848-033d-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-25T19:52:44.587 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:44 smithi132 systemd[1]: /etc/systemd/system/ceph-3114c848-033d-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-25T19:52:44.771 INFO:teuthology.orchestra.run.smithi132.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-3114c848-033d-11ef-bc93-c7b262605968.target.wants/ceph-3114c848-033d-11ef-bc93-c7b262605968@mgr.x.service -> /etc/systemd/system/ceph-3114c848-033d-11ef-bc93-c7b262605968@.service. 2024-04-25T19:52:44.778 INFO:teuthology.orchestra.run.smithi132.stdout:firewalld does not appear to be present 2024-04-25T19:52:44.778 INFO:teuthology.orchestra.run.smithi132.stdout:Not possible to enable service . firewalld.service is not available 2024-04-25T19:52:44.778 INFO:teuthology.orchestra.run.smithi132.stdout:firewalld does not appear to be present 2024-04-25T19:52:44.778 INFO:teuthology.orchestra.run.smithi132.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-25T19:52:44.778 INFO:teuthology.orchestra.run.smithi132.stdout:Waiting for mgr to start... 2024-04-25T19:52:44.778 INFO:teuthology.orchestra.run.smithi132.stdout:Waiting for mgr... 2024-04-25T19:52:45.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:44 smithi132 bash[18607]: audit 2024-04-25T19:52:43.592060+0000 mon.a (mon.0) 12 : audit [INF] from='client.? 172.21.15.132:0/1582245656' entity='client.admin' 2024-04-25T19:52:45.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:44 smithi132 bash[18607]: audit 2024-04-25T19:52:43.592060+0000 mon.a (mon.0) 12 : audit [INF] from='client.? 172.21.15.132:0/1582245656' entity='client.admin' 2024-04-25T19:52:45.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:44 smithi132 systemd[1]: /etc/systemd/system/ceph-3114c848-033d-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-25T19:52:46.440 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:46 smithi132 bash[18607]: audit 2024-04-25T19:52:46.229233+0000 mon.a (mon.0) 13 : audit [DBG] from='client.? 172.21.15.132:0/942175447' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T19:52:46.440 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:46 smithi132 bash[18607]: audit 2024-04-25T19:52:46.229233+0000 mon.a (mon.0) 13 : audit [DBG] from='client.? 172.21.15.132:0/942175447' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T19:52:46.440 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:52:46 smithi132 bash[18860]: debug 2024-04-25T19:52:46.189+0000 7f489eed7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T19:52:46.833 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout 2024-04-25T19:52:46.833 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout { 2024-04-25T19:52:46.833 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "fsid": "3114c848-033d-11ef-bc93-c7b262605968", 2024-04-25T19:52:46.833 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T19:52:46.833 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T19:52:46.833 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T19:52:46.834 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T19:52:46.834 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-04-25T19:52:46.834 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T19:52:46.834 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T19:52:46.834 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout 0 2024-04-25T19:52:46.834 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout ], 2024-04-25T19:52:46.834 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T19:52:46.834 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T19:52:46.834 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout ], 2024-04-25T19:52:46.834 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-25T19:52:46.834 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T19:52:46.834 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T19:52:46.835 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T19:52:46.835 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T19:52:46.835 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-04-25T19:52:46.835 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T19:52:46.835 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T19:52:46.835 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T19:52:46.835 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T19:52:46.835 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T19:52:46.835 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T19:52:46.835 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T19:52:46.835 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T19:52:46.835 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-04-25T19:52:46.835 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T19:52:46.835 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T19:52:46.835 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T19:52:46.836 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T19:52:46.836 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T19:52:46.836 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T19:52:46.836 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T19:52:46.836 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T19:52:46.836 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T19:52:46.836 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-04-25T19:52:46.836 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T19:52:46.836 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T19:52:46.836 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T19:52:46.836 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T19:52:46.836 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-04-25T19:52:46.836 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T19:52:46.836 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-25T19:52:46.837 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T19:52:46.837 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T19:52:46.837 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T19:52:46.837 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T19:52:46.837 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T19:52:46.837 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout ], 2024-04-25T19:52:46.837 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T19:52:46.837 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-04-25T19:52:46.837 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T19:52:46.837 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T19:52:46.837 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T19:52:37.558638+0000", 2024-04-25T19:52:46.837 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T19:52:46.837 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-04-25T19:52:46.837 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T19:52:46.838 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout } 2024-04-25T19:52:46.838 INFO:teuthology.orchestra.run.smithi132.stdout:mgr not available, waiting (1/15)... 2024-04-25T19:52:48.021 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:52:47 smithi132 bash[18860]: debug 2024-04-25T19:52:47.745+0000 7f489eed7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T19:52:48.021 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:52:47 smithi132 bash[18860]: debug 2024-04-25T19:52:47.841+0000 7f489eed7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T19:52:48.520 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:52:48 smithi132 bash[18860]: debug 2024-04-25T19:52:48.089+0000 7f489eed7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T19:52:49.270 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:52:48 smithi132 bash[18860]: debug 2024-04-25T19:52:48.881+0000 7f489eed7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T19:52:49.271 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:52:48 smithi132 bash[18860]: debug 2024-04-25T19:52:48.989+0000 7f489eed7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T19:52:49.271 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:52:49 smithi132 bash[18860]: debug 2024-04-25T19:52:49.089+0000 7f489eed7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T19:52:49.770 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:52:49 smithi132 bash[18860]: debug 2024-04-25T19:52:49.445+0000 7f489eed7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T19:52:49.771 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:52:49 smithi132 bash[18860]: debug 2024-04-25T19:52:49.557+0000 7f489eed7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T19:52:49.771 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:52:49 smithi132 bash[18860]: debug 2024-04-25T19:52:49.661+0000 7f489eed7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T19:52:50.265 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:49 smithi132 bash[18607]: audit 2024-04-25T19:52:49.920667+0000 mon.a (mon.0) 14 : audit [DBG] from='client.? 172.21.15.132:0/1591906964' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T19:52:50.265 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:49 smithi132 bash[18607]: audit 2024-04-25T19:52:49.920667+0000 mon.a (mon.0) 14 : audit [DBG] from='client.? 172.21.15.132:0/1591906964' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T19:52:50.266 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:52:50 smithi132 bash[18860]: debug 2024-04-25T19:52:50.021+0000 7f489eed7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T19:52:50.520 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:52:50 smithi132 bash[18860]: debug 2024-04-25T19:52:50.265+0000 7f489eed7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T19:52:50.542 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout 2024-04-25T19:52:50.542 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout { 2024-04-25T19:52:50.542 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "fsid": "3114c848-033d-11ef-bc93-c7b262605968", 2024-04-25T19:52:50.542 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T19:52:50.542 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T19:52:50.542 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T19:52:50.542 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T19:52:50.543 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-04-25T19:52:50.543 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T19:52:50.543 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T19:52:50.543 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout 0 2024-04-25T19:52:50.543 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout ], 2024-04-25T19:52:50.543 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T19:52:50.543 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T19:52:50.543 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout ], 2024-04-25T19:52:50.543 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-04-25T19:52:50.543 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T19:52:50.543 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T19:52:50.543 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T19:52:50.543 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T19:52:50.543 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-04-25T19:52:50.544 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T19:52:50.544 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T19:52:50.544 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T19:52:50.544 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T19:52:50.544 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T19:52:50.544 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T19:52:50.544 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T19:52:50.544 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T19:52:50.547 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-04-25T19:52:50.547 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T19:52:50.547 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T19:52:50.547 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T19:52:50.548 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T19:52:50.548 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T19:52:50.548 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T19:52:50.548 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T19:52:50.548 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T19:52:50.548 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T19:52:50.548 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-04-25T19:52:50.548 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T19:52:50.548 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T19:52:50.548 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T19:52:50.548 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T19:52:50.548 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-04-25T19:52:50.548 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T19:52:50.548 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-25T19:52:50.549 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T19:52:50.549 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T19:52:50.549 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T19:52:50.549 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T19:52:50.549 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T19:52:50.549 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout ], 2024-04-25T19:52:50.549 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T19:52:50.549 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-04-25T19:52:50.549 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T19:52:50.549 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T19:52:50.549 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T19:52:37.558638+0000", 2024-04-25T19:52:50.549 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T19:52:50.549 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-04-25T19:52:50.549 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T19:52:50.550 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout } 2024-04-25T19:52:50.550 INFO:teuthology.orchestra.run.smithi132.stdout:mgr not available, waiting (2/15)... 2024-04-25T19:52:50.909 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:52:50 smithi132 bash[18860]: debug 2024-04-25T19:52:50.525+0000 7f489eed7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T19:52:50.909 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:52:50 smithi132 bash[18860]: debug 2024-04-25T19:52:50.637+0000 7f489eed7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T19:52:50.909 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:52:50 smithi132 bash[18860]: debug 2024-04-25T19:52:50.905+0000 7f489eed7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T19:52:51.671 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:52:51 smithi132 bash[18860]: debug 2024-04-25T19:52:51.317+0000 7f489eed7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T19:52:51.671 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:52:51 smithi132 bash[18860]: debug 2024-04-25T19:52:51.553+0000 7f489eed7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T19:52:51.671 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:52:51 smithi132 bash[18860]: debug 2024-04-25T19:52:51.669+0000 7f489eed7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T19:52:52.020 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:52:51 smithi132 bash[18860]: debug 2024-04-25T19:52:51.909+0000 7f489eed7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T19:52:53.260 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:52:52 smithi132 bash[18860]: debug 2024-04-25T19:52:52.913+0000 7f489eed7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T19:52:53.261 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:52:53 smithi132 bash[18860]: debug 2024-04-25T19:52:53.029+0000 7f489eed7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T19:52:53.520 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:52:53 smithi132 bash[18860]: debug 2024-04-25T19:52:53.325+0000 7f489eed7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T19:52:53.521 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:52:53 smithi132 bash[18860]: debug 2024-04-25T19:52:53.441+0000 7f489eed7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T19:52:54.021 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:52:53 smithi132 bash[18860]: debug 2024-04-25T19:52:53.721+0000 7f489eed7200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-04-25T19:52:54.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:53 smithi132 bash[18607]: audit 2024-04-25T19:52:53.594953+0000 mon.a (mon.0) 15 : audit [DBG] from='client.? 172.21.15.132:0/3180767559' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T19:52:54.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:53 smithi132 bash[18607]: audit 2024-04-25T19:52:53.594953+0000 mon.a (mon.0) 15 : audit [DBG] from='client.? 172.21.15.132:0/3180767559' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T19:52:54.129 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout 2024-04-25T19:52:54.129 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout { 2024-04-25T19:52:54.129 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "fsid": "3114c848-033d-11ef-bc93-c7b262605968", 2024-04-25T19:52:54.129 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T19:52:54.129 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T19:52:54.129 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T19:52:54.129 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T19:52:54.129 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-04-25T19:52:54.130 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T19:52:54.130 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T19:52:54.130 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout 0 2024-04-25T19:52:54.130 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout ], 2024-04-25T19:52:54.130 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T19:52:54.130 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T19:52:54.130 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout ], 2024-04-25T19:52:54.130 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-04-25T19:52:54.130 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T19:52:54.131 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T19:52:54.131 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T19:52:54.131 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T19:52:54.131 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-04-25T19:52:54.131 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T19:52:54.131 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T19:52:54.131 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T19:52:54.131 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T19:52:54.131 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T19:52:54.131 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T19:52:54.131 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T19:52:54.131 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T19:52:54.135 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-04-25T19:52:54.135 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T19:52:54.135 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T19:52:54.135 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T19:52:54.135 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T19:52:54.135 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T19:52:54.135 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T19:52:54.135 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T19:52:54.135 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T19:52:54.135 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T19:52:54.136 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-04-25T19:52:54.136 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T19:52:54.136 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T19:52:54.136 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T19:52:54.136 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T19:52:54.136 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-04-25T19:52:54.136 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T19:52:54.136 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-25T19:52:54.136 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T19:52:54.136 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T19:52:54.136 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T19:52:54.136 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T19:52:54.137 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T19:52:54.137 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout ], 2024-04-25T19:52:54.137 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T19:52:54.137 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-04-25T19:52:54.137 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T19:52:54.137 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T19:52:54.137 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T19:52:37.558638+0000", 2024-04-25T19:52:54.137 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T19:52:54.137 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-04-25T19:52:54.137 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T19:52:54.137 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout } 2024-04-25T19:52:54.138 INFO:teuthology.orchestra.run.smithi132.stdout:mgr not available, waiting (3/15)... 2024-04-25T19:52:54.520 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:52:54 smithi132 bash[18860]: debug 2024-04-25T19:52:54.173+0000 7f489eed7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T19:52:54.521 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:52:54 smithi132 bash[18860]: debug 2024-04-25T19:52:54.293+0000 7f489eed7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T19:52:55.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:54 smithi132 bash[18607]: cluster 2024-04-25T19:52:54.319784+0000 mon.a (mon.0) 16 : cluster [INF] Activating manager daemon x 2024-04-25T19:52:55.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:54 smithi132 bash[18607]: cluster 2024-04-25T19:52:54.319784+0000 mon.a (mon.0) 16 : cluster [INF] Activating manager daemon x 2024-04-25T19:52:55.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:54 smithi132 bash[18607]: cluster 2024-04-25T19:52:54.326653+0000 mon.a (mon.0) 17 : cluster [DBG] mgrmap e2: x(active, starting, since 0.00718718s) 2024-04-25T19:52:55.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:54 smithi132 bash[18607]: cluster 2024-04-25T19:52:54.326653+0000 mon.a (mon.0) 17 : cluster [DBG] mgrmap e2: x(active, starting, since 0.00718718s) 2024-04-25T19:52:55.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:54 smithi132 bash[18607]: audit 2024-04-25T19:52:54.327794+0000 mon.a (mon.0) 18 : audit [DBG] from='mgr.14100 172.21.15.132:0/2546841058' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T19:52:55.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:54 smithi132 bash[18607]: audit 2024-04-25T19:52:54.327794+0000 mon.a (mon.0) 18 : audit [DBG] from='mgr.14100 172.21.15.132:0/2546841058' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T19:52:55.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:54 smithi132 bash[18607]: audit 2024-04-25T19:52:54.327993+0000 mon.a (mon.0) 19 : audit [DBG] from='mgr.14100 172.21.15.132:0/2546841058' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T19:52:55.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:54 smithi132 bash[18607]: audit 2024-04-25T19:52:54.327993+0000 mon.a (mon.0) 19 : audit [DBG] from='mgr.14100 172.21.15.132:0/2546841058' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T19:52:55.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:54 smithi132 bash[18607]: audit 2024-04-25T19:52:54.328111+0000 mon.a (mon.0) 20 : audit [DBG] from='mgr.14100 172.21.15.132:0/2546841058' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T19:52:55.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:54 smithi132 bash[18607]: audit 2024-04-25T19:52:54.328111+0000 mon.a (mon.0) 20 : audit [DBG] from='mgr.14100 172.21.15.132:0/2546841058' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T19:52:55.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:54 smithi132 bash[18607]: audit 2024-04-25T19:52:54.328231+0000 mon.a (mon.0) 21 : audit [DBG] from='mgr.14100 172.21.15.132:0/2546841058' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T19:52:55.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:54 smithi132 bash[18607]: audit 2024-04-25T19:52:54.328231+0000 mon.a (mon.0) 21 : audit [DBG] from='mgr.14100 172.21.15.132:0/2546841058' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T19:52:55.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:54 smithi132 bash[18607]: audit 2024-04-25T19:52:54.328737+0000 mon.a (mon.0) 22 : audit [DBG] from='mgr.14100 172.21.15.132:0/2546841058' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T19:52:55.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:54 smithi132 bash[18607]: audit 2024-04-25T19:52:54.328737+0000 mon.a (mon.0) 22 : audit [DBG] from='mgr.14100 172.21.15.132:0/2546841058' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T19:52:55.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:54 smithi132 bash[18607]: cluster 2024-04-25T19:52:54.354468+0000 mon.a (mon.0) 23 : cluster [INF] Manager daemon x is now available 2024-04-25T19:52:55.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:54 smithi132 bash[18607]: cluster 2024-04-25T19:52:54.354468+0000 mon.a (mon.0) 23 : cluster [INF] Manager daemon x is now available 2024-04-25T19:52:55.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:54 smithi132 bash[18607]: audit 2024-04-25T19:52:54.366282+0000 mon.a (mon.0) 24 : audit [INF] from='mgr.14100 172.21.15.132:0/2546841058' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T19:52:55.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:54 smithi132 bash[18607]: audit 2024-04-25T19:52:54.366282+0000 mon.a (mon.0) 24 : audit [INF] from='mgr.14100 172.21.15.132:0/2546841058' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T19:52:55.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:54 smithi132 bash[18607]: audit 2024-04-25T19:52:54.369254+0000 mon.a (mon.0) 25 : audit [INF] from='mgr.14100 172.21.15.132:0/2546841058' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T19:52:55.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:54 smithi132 bash[18607]: audit 2024-04-25T19:52:54.369254+0000 mon.a (mon.0) 25 : audit [INF] from='mgr.14100 172.21.15.132:0/2546841058' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T19:52:55.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:54 smithi132 bash[18607]: audit 2024-04-25T19:52:54.372410+0000 mon.a (mon.0) 26 : audit [INF] from='mgr.14100 172.21.15.132:0/2546841058' entity='mgr.x' 2024-04-25T19:52:55.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:54 smithi132 bash[18607]: audit 2024-04-25T19:52:54.372410+0000 mon.a (mon.0) 26 : audit [INF] from='mgr.14100 172.21.15.132:0/2546841058' entity='mgr.x' 2024-04-25T19:52:55.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:54 smithi132 bash[18607]: audit 2024-04-25T19:52:54.375468+0000 mon.a (mon.0) 27 : audit [INF] from='mgr.14100 172.21.15.132:0/2546841058' entity='mgr.x' 2024-04-25T19:52:55.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:54 smithi132 bash[18607]: audit 2024-04-25T19:52:54.375468+0000 mon.a (mon.0) 27 : audit [INF] from='mgr.14100 172.21.15.132:0/2546841058' entity='mgr.x' 2024-04-25T19:52:55.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:54 smithi132 bash[18607]: audit 2024-04-25T19:52:54.381050+0000 mon.a (mon.0) 28 : audit [INF] from='mgr.14100 172.21.15.132:0/2546841058' entity='mgr.x' 2024-04-25T19:52:55.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:54 smithi132 bash[18607]: audit 2024-04-25T19:52:54.381050+0000 mon.a (mon.0) 28 : audit [INF] from='mgr.14100 172.21.15.132:0/2546841058' entity='mgr.x' 2024-04-25T19:52:56.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:56 smithi132 bash[18607]: cluster 2024-04-25T19:52:55.335438+0000 mon.a (mon.0) 29 : cluster [DBG] mgrmap e3: x(active, since 1.01597s) 2024-04-25T19:52:56.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:56 smithi132 bash[18607]: cluster 2024-04-25T19:52:55.335438+0000 mon.a (mon.0) 29 : cluster [DBG] mgrmap e3: x(active, since 1.01597s) 2024-04-25T19:52:58.235 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout 2024-04-25T19:52:58.236 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout { 2024-04-25T19:52:58.236 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "fsid": "3114c848-033d-11ef-bc93-c7b262605968", 2024-04-25T19:52:58.236 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T19:52:58.236 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T19:52:58.236 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T19:52:58.236 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T19:52:58.236 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-04-25T19:52:58.236 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T19:52:58.236 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T19:52:58.236 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout 0 2024-04-25T19:52:58.236 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout ], 2024-04-25T19:52:58.236 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T19:52:58.237 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T19:52:58.237 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout ], 2024-04-25T19:52:58.237 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-04-25T19:52:58.237 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T19:52:58.237 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T19:52:58.237 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T19:52:58.237 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T19:52:58.237 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-04-25T19:52:58.238 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T19:52:58.238 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T19:52:58.238 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T19:52:58.238 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T19:52:58.238 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T19:52:58.238 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T19:52:58.238 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T19:52:58.238 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T19:52:58.241 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-04-25T19:52:58.241 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T19:52:58.241 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T19:52:58.242 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T19:52:58.242 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T19:52:58.242 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T19:52:58.242 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T19:52:58.242 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T19:52:58.242 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T19:52:58.242 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T19:52:58.242 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-04-25T19:52:58.242 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T19:52:58.242 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T19:52:58.242 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T19:52:58.242 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T19:52:58.242 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-04-25T19:52:58.243 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T19:52:58.243 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T19:52:58.243 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T19:52:58.243 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T19:52:58.243 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T19:52:58.243 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T19:52:58.243 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T19:52:58.243 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout ], 2024-04-25T19:52:58.243 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T19:52:58.243 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-04-25T19:52:58.243 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T19:52:58.243 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T19:52:58.243 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T19:52:37.558638+0000", 2024-04-25T19:52:58.243 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T19:52:58.244 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout }, 2024-04-25T19:52:58.244 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T19:52:58.244 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout } 2024-04-25T19:52:58.244 INFO:teuthology.orchestra.run.smithi132.stdout:mgr is available 2024-04-25T19:52:58.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:58 smithi132 bash[18607]: cluster 2024-04-25T19:52:57.333249+0000 mon.a (mon.0) 30 : cluster [DBG] mgrmap e4: x(active, since 3s) 2024-04-25T19:52:58.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:58 smithi132 bash[18607]: cluster 2024-04-25T19:52:57.333249+0000 mon.a (mon.0) 30 : cluster [DBG] mgrmap e4: x(active, since 3s) 2024-04-25T19:52:58.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:58 smithi132 bash[18607]: audit 2024-04-25T19:52:57.703287+0000 mon.a (mon.0) 31 : audit [DBG] from='client.? 172.21.15.132:0/2464855385' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T19:52:58.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:58 smithi132 bash[18607]: audit 2024-04-25T19:52:57.703287+0000 mon.a (mon.0) 31 : audit [DBG] from='client.? 172.21.15.132:0/2464855385' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T19:53:00.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:59 smithi132 bash[18607]: audit 2024-04-25T19:52:59.570523+0000 mon.a (mon.0) 32 : audit [INF] from='client.? 172.21.15.132:0/521078263' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-25T19:53:00.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:52:59 smithi132 bash[18607]: audit 2024-04-25T19:52:59.570523+0000 mon.a (mon.0) 32 : audit [INF] from='client.? 172.21.15.132:0/521078263' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-25T19:53:00.220 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout 2024-04-25T19:53:00.220 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout [global] 2024-04-25T19:53:00.220 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout fsid = 3114c848-033d-11ef-bc93-c7b262605968 2024-04-25T19:53:00.220 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-25T19:53:00.220 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-25T19:53:00.220 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-25T19:53:00.220 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-25T19:53:00.220 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-25T19:53:00.220 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout 2024-04-25T19:53:00.220 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-25T19:53:00.221 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-25T19:53:00.221 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout 2024-04-25T19:53:00.221 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout [osd] 2024-04-25T19:53:00.221 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-25T19:53:00.221 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-25T19:53:00.221 INFO:teuthology.orchestra.run.smithi132.stdout:Enabling cephadm module... 2024-04-25T19:53:02.020 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:01 smithi132 bash[18860]: ignoring --setuser ceph since I am not root 2024-04-25T19:53:02.021 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:01 smithi132 bash[18860]: ignoring --setgroup ceph since I am not root 2024-04-25T19:53:02.021 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:01 smithi132 bash[18860]: debug 2024-04-25T19:53:01.781+0000 7fa113c13200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T19:53:02.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:01 smithi132 bash[18607]: audit 2024-04-25T19:53:01.542697+0000 mon.a (mon.0) 33 : audit [INF] from='client.? 172.21.15.132:0/4102731973' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-25T19:53:02.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:01 smithi132 bash[18607]: audit 2024-04-25T19:53:01.542697+0000 mon.a (mon.0) 33 : audit [INF] from='client.? 172.21.15.132:0/4102731973' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-25T19:53:03.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:02 smithi132 bash[18607]: audit 2024-04-25T19:53:01.594633+0000 mon.a (mon.0) 34 : audit [INF] from='client.? 172.21.15.132:0/4102731973' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-25T19:53:03.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:02 smithi132 bash[18607]: audit 2024-04-25T19:53:01.594633+0000 mon.a (mon.0) 34 : audit [INF] from='client.? 172.21.15.132:0/4102731973' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-25T19:53:03.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:02 smithi132 bash[18607]: cluster 2024-04-25T19:53:01.599895+0000 mon.a (mon.0) 35 : cluster [DBG] mgrmap e5: x(active, since 7s) 2024-04-25T19:53:03.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:02 smithi132 bash[18607]: cluster 2024-04-25T19:53:01.599895+0000 mon.a (mon.0) 35 : cluster [DBG] mgrmap e5: x(active, since 7s) 2024-04-25T19:53:03.636 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:03 smithi132 bash[18860]: debug 2024-04-25T19:53:03.357+0000 7fa113c13200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T19:53:03.636 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:03 smithi132 bash[18860]: debug 2024-04-25T19:53:03.453+0000 7fa113c13200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T19:53:03.636 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:03 smithi132 bash[18607]: audit 2024-04-25T19:53:03.457775+0000 mon.a (mon.0) 36 : audit [DBG] from='client.? 172.21.15.132:0/3628353295' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-25T19:53:03.636 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:03 smithi132 bash[18607]: audit 2024-04-25T19:53:03.457775+0000 mon.a (mon.0) 36 : audit [DBG] from='client.? 172.21.15.132:0/3628353295' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-25T19:53:03.972 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout { 2024-04-25T19:53:03.972 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-25T19:53:03.973 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T19:53:03.973 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-25T19:53:03.973 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-25T19:53:03.973 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout } 2024-04-25T19:53:03.973 INFO:teuthology.orchestra.run.smithi132.stdout:Waiting for the mgr to restart... 2024-04-25T19:53:03.973 INFO:teuthology.orchestra.run.smithi132.stdout:Waiting for mgr epoch 5... 2024-04-25T19:53:03.989 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:03 smithi132 bash[18860]: debug 2024-04-25T19:53:03.709+0000 7fa113c13200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T19:53:04.770 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:04 smithi132 bash[18860]: debug 2024-04-25T19:53:04.509+0000 7fa113c13200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T19:53:04.770 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:04 smithi132 bash[18860]: debug 2024-04-25T19:53:04.621+0000 7fa113c13200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T19:53:04.771 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:04 smithi132 bash[18860]: debug 2024-04-25T19:53:04.729+0000 7fa113c13200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T19:53:05.520 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:05 smithi132 bash[18860]: debug 2024-04-25T19:53:05.089+0000 7fa113c13200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T19:53:05.520 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:05 smithi132 bash[18860]: debug 2024-04-25T19:53:05.201+0000 7fa113c13200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T19:53:05.521 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:05 smithi132 bash[18860]: debug 2024-04-25T19:53:05.305+0000 7fa113c13200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T19:53:06.020 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:05 smithi132 bash[18860]: debug 2024-04-25T19:53:05.657+0000 7fa113c13200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T19:53:06.020 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:05 smithi132 bash[18860]: debug 2024-04-25T19:53:05.897+0000 7fa113c13200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T19:53:06.520 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:06 smithi132 bash[18860]: debug 2024-04-25T19:53:06.157+0000 7fa113c13200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T19:53:06.521 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:06 smithi132 bash[18860]: debug 2024-04-25T19:53:06.269+0000 7fa113c13200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T19:53:06.954 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:06 smithi132 bash[18860]: debug 2024-04-25T19:53:06.541+0000 7fa113c13200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T19:53:07.270 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:06 smithi132 bash[18860]: debug 2024-04-25T19:53:06.953+0000 7fa113c13200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T19:53:07.270 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:07 smithi132 bash[18860]: debug 2024-04-25T19:53:07.189+0000 7fa113c13200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T19:53:07.770 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:07 smithi132 bash[18860]: debug 2024-04-25T19:53:07.305+0000 7fa113c13200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T19:53:07.770 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:07 smithi132 bash[18860]: debug 2024-04-25T19:53:07.549+0000 7fa113c13200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T19:53:08.965 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:08 smithi132 bash[18860]: debug 2024-04-25T19:53:08.557+0000 7fa113c13200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T19:53:08.965 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:08 smithi132 bash[18860]: debug 2024-04-25T19:53:08.673+0000 7fa113c13200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T19:53:09.270 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:08 smithi132 bash[18860]: debug 2024-04-25T19:53:08.965+0000 7fa113c13200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T19:53:09.270 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:09 smithi132 bash[18860]: debug 2024-04-25T19:53:09.077+0000 7fa113c13200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T19:53:09.770 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:09 smithi132 bash[18860]: debug 2024-04-25T19:53:09.353+0000 7fa113c13200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-04-25T19:53:10.271 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:09 smithi132 bash[18860]: debug 2024-04-25T19:53:09.805+0000 7fa113c13200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T19:53:10.271 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:09 smithi132 bash[18860]: debug 2024-04-25T19:53:09.925+0000 7fa113c13200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T19:53:10.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:10 smithi132 bash[18607]: cluster 2024-04-25T19:53:09.950835+0000 mon.a (mon.0) 37 : cluster [INF] Active manager daemon x restarted 2024-04-25T19:53:10.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:10 smithi132 bash[18607]: cluster 2024-04-25T19:53:09.950835+0000 mon.a (mon.0) 37 : cluster [INF] Active manager daemon x restarted 2024-04-25T19:53:10.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:10 smithi132 bash[18607]: cluster 2024-04-25T19:53:09.951324+0000 mon.a (mon.0) 38 : cluster [INF] Activating manager daemon x 2024-04-25T19:53:10.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:10 smithi132 bash[18607]: cluster 2024-04-25T19:53:09.951324+0000 mon.a (mon.0) 38 : cluster [INF] Activating manager daemon x 2024-04-25T19:53:10.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:10 smithi132 bash[18607]: cluster 2024-04-25T19:53:09.960475+0000 mon.a (mon.0) 39 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-04-25T19:53:10.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:10 smithi132 bash[18607]: cluster 2024-04-25T19:53:09.960475+0000 mon.a (mon.0) 39 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-04-25T19:53:10.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:10 smithi132 bash[18607]: cluster 2024-04-25T19:53:09.960775+0000 mon.a (mon.0) 40 : cluster [DBG] mgrmap e6: x(active, starting, since 0.00963432s) 2024-04-25T19:53:10.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:10 smithi132 bash[18607]: cluster 2024-04-25T19:53:09.960775+0000 mon.a (mon.0) 40 : cluster [DBG] mgrmap e6: x(active, starting, since 0.00963432s) 2024-04-25T19:53:10.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:10 smithi132 bash[18607]: audit 2024-04-25T19:53:09.962755+0000 mon.a (mon.0) 41 : audit [DBG] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T19:53:10.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:10 smithi132 bash[18607]: audit 2024-04-25T19:53:09.962755+0000 mon.a (mon.0) 41 : audit [DBG] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T19:53:10.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:10 smithi132 bash[18607]: audit 2024-04-25T19:53:09.963841+0000 mon.a (mon.0) 42 : audit [DBG] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T19:53:10.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:10 smithi132 bash[18607]: audit 2024-04-25T19:53:09.963841+0000 mon.a (mon.0) 42 : audit [DBG] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T19:53:10.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:10 smithi132 bash[18607]: audit 2024-04-25T19:53:09.964993+0000 mon.a (mon.0) 43 : audit [DBG] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T19:53:10.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:10 smithi132 bash[18607]: audit 2024-04-25T19:53:09.964993+0000 mon.a (mon.0) 43 : audit [DBG] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T19:53:10.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:10 smithi132 bash[18607]: audit 2024-04-25T19:53:09.965210+0000 mon.a (mon.0) 44 : audit [DBG] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T19:53:10.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:10 smithi132 bash[18607]: audit 2024-04-25T19:53:09.965210+0000 mon.a (mon.0) 44 : audit [DBG] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T19:53:10.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:10 smithi132 bash[18607]: audit 2024-04-25T19:53:09.965389+0000 mon.a (mon.0) 45 : audit [DBG] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T19:53:10.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:10 smithi132 bash[18607]: audit 2024-04-25T19:53:09.965389+0000 mon.a (mon.0) 45 : audit [DBG] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T19:53:10.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:10 smithi132 bash[18607]: cluster 2024-04-25T19:53:09.990781+0000 mon.a (mon.0) 46 : cluster [INF] Manager daemon x is now available 2024-04-25T19:53:10.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:10 smithi132 bash[18607]: cluster 2024-04-25T19:53:09.990781+0000 mon.a (mon.0) 46 : cluster [INF] Manager daemon x is now available 2024-04-25T19:53:11.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:11 smithi132 bash[18607]: cephadm 2024-04-25T19:53:10.011065+0000 mgr.x (mgr.14120) 1 : cephadm [INF] Found migration_current of "None". Setting to last migration. 2024-04-25T19:53:11.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:11 smithi132 bash[18607]: cephadm 2024-04-25T19:53:10.011065+0000 mgr.x (mgr.14120) 1 : cephadm [INF] Found migration_current of "None". Setting to last migration. 2024-04-25T19:53:11.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:11 smithi132 bash[18607]: audit 2024-04-25T19:53:10.013699+0000 mon.a (mon.0) 47 : audit [INF] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' 2024-04-25T19:53:11.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:11 smithi132 bash[18607]: audit 2024-04-25T19:53:10.013699+0000 mon.a (mon.0) 47 : audit [INF] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' 2024-04-25T19:53:11.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:11 smithi132 bash[18607]: audit 2024-04-25T19:53:10.019151+0000 mon.a (mon.0) 48 : audit [INF] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' 2024-04-25T19:53:11.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:11 smithi132 bash[18607]: audit 2024-04-25T19:53:10.019151+0000 mon.a (mon.0) 48 : audit [INF] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' 2024-04-25T19:53:11.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:11 smithi132 bash[18607]: audit 2024-04-25T19:53:10.023416+0000 mon.a (mon.0) 49 : audit [DBG] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:53:11.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:11 smithi132 bash[18607]: audit 2024-04-25T19:53:10.023416+0000 mon.a (mon.0) 49 : audit [DBG] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:53:11.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:11 smithi132 bash[18607]: audit 2024-04-25T19:53:10.025484+0000 mon.a (mon.0) 50 : audit [DBG] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:53:11.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:11 smithi132 bash[18607]: audit 2024-04-25T19:53:10.025484+0000 mon.a (mon.0) 50 : audit [DBG] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:53:11.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:11 smithi132 bash[18607]: audit 2024-04-25T19:53:10.037740+0000 mon.a (mon.0) 51 : audit [INF] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T19:53:11.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:11 smithi132 bash[18607]: audit 2024-04-25T19:53:10.037740+0000 mon.a (mon.0) 51 : audit [INF] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T19:53:11.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:11 smithi132 bash[18607]: audit 2024-04-25T19:53:10.040956+0000 mon.a (mon.0) 52 : audit [INF] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T19:53:11.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:11 smithi132 bash[18607]: audit 2024-04-25T19:53:10.040956+0000 mon.a (mon.0) 52 : audit [INF] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T19:53:11.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:11 smithi132 bash[18607]: audit 2024-04-25T19:53:10.359470+0000 mon.a (mon.0) 53 : audit [INF] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' 2024-04-25T19:53:11.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:11 smithi132 bash[18607]: audit 2024-04-25T19:53:10.359470+0000 mon.a (mon.0) 53 : audit [INF] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' 2024-04-25T19:53:11.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:11 smithi132 bash[18607]: audit 2024-04-25T19:53:10.365918+0000 mon.a (mon.0) 54 : audit [INF] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' 2024-04-25T19:53:11.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:11 smithi132 bash[18607]: audit 2024-04-25T19:53:10.365918+0000 mon.a (mon.0) 54 : audit [INF] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' 2024-04-25T19:53:11.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:11 smithi132 bash[18607]: cluster 2024-04-25T19:53:10.962557+0000 mon.a (mon.0) 55 : cluster [DBG] mgrmap e7: x(active, since 1.01142s) 2024-04-25T19:53:11.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:11 smithi132 bash[18607]: cluster 2024-04-25T19:53:10.962557+0000 mon.a (mon.0) 55 : cluster [DBG] mgrmap e7: x(active, since 1.01142s) 2024-04-25T19:53:11.588 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout { 2024-04-25T19:53:11.588 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-25T19:53:11.588 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-25T19:53:11.588 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout } 2024-04-25T19:53:11.588 INFO:teuthology.orchestra.run.smithi132.stdout:mgr epoch 5 is available 2024-04-25T19:53:11.588 INFO:teuthology.orchestra.run.smithi132.stdout:Setting orchestrator backend to cephadm... 2024-04-25T19:53:12.012 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:12 smithi132 bash[18607]: cephadm 2024-04-25T19:53:10.875339+0000 mgr.x (mgr.14120) 2 : cephadm [INF] [25/Apr/2024:19:53:10] ENGINE Bus STARTING 2024-04-25T19:53:12.012 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:12 smithi132 bash[18607]: cephadm 2024-04-25T19:53:10.875339+0000 mgr.x (mgr.14120) 2 : cephadm [INF] [25/Apr/2024:19:53:10] ENGINE Bus STARTING 2024-04-25T19:53:12.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:12 smithi132 bash[18607]: audit 2024-04-25T19:53:10.962203+0000 mgr.x (mgr.14120) 3 : audit [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-25T19:53:12.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:12 smithi132 bash[18607]: audit 2024-04-25T19:53:10.962203+0000 mgr.x (mgr.14120) 3 : audit [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-25T19:53:12.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:12 smithi132 bash[18607]: cephadm 2024-04-25T19:53:10.977519+0000 mgr.x (mgr.14120) 4 : cephadm [INF] [25/Apr/2024:19:53:10] ENGINE Serving on http://172.21.15.132:8765 2024-04-25T19:53:12.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:12 smithi132 bash[18607]: cephadm 2024-04-25T19:53:10.977519+0000 mgr.x (mgr.14120) 4 : cephadm [INF] [25/Apr/2024:19:53:10] ENGINE Serving on http://172.21.15.132:8765 2024-04-25T19:53:12.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:12 smithi132 bash[18607]: audit 2024-04-25T19:53:10.978378+0000 mgr.x (mgr.14120) 5 : audit [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-25T19:53:12.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:12 smithi132 bash[18607]: audit 2024-04-25T19:53:10.978378+0000 mgr.x (mgr.14120) 5 : audit [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-25T19:53:12.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:12 smithi132 bash[18607]: cephadm 2024-04-25T19:53:11.095808+0000 mgr.x (mgr.14120) 6 : cephadm [INF] [25/Apr/2024:19:53:11] ENGINE Serving on https://172.21.15.132:7150 2024-04-25T19:53:12.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:12 smithi132 bash[18607]: cephadm 2024-04-25T19:53:11.095808+0000 mgr.x (mgr.14120) 6 : cephadm [INF] [25/Apr/2024:19:53:11] ENGINE Serving on https://172.21.15.132:7150 2024-04-25T19:53:12.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:12 smithi132 bash[18607]: cephadm 2024-04-25T19:53:11.096082+0000 mgr.x (mgr.14120) 7 : cephadm [INF] [25/Apr/2024:19:53:11] ENGINE Bus STARTED 2024-04-25T19:53:12.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:12 smithi132 bash[18607]: cephadm 2024-04-25T19:53:11.096082+0000 mgr.x (mgr.14120) 7 : cephadm [INF] [25/Apr/2024:19:53:11] ENGINE Bus STARTED 2024-04-25T19:53:12.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:12 smithi132 bash[18607]: audit 2024-04-25T19:53:11.145631+0000 mon.a (mon.0) 56 : audit [DBG] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:53:12.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:12 smithi132 bash[18607]: audit 2024-04-25T19:53:11.145631+0000 mon.a (mon.0) 56 : audit [DBG] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:53:13.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:13 smithi132 bash[18607]: cluster 2024-04-25T19:53:12.028479+0000 mon.a (mon.0) 57 : cluster [DBG] mgrmap e8: x(active, since 2s) 2024-04-25T19:53:13.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:13 smithi132 bash[18607]: cluster 2024-04-25T19:53:12.028479+0000 mon.a (mon.0) 57 : cluster [DBG] mgrmap e8: x(active, since 2s) 2024-04-25T19:53:13.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:13 smithi132 bash[18607]: audit 2024-04-25T19:53:12.821244+0000 mon.a (mon.0) 58 : audit [INF] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' 2024-04-25T19:53:13.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:13 smithi132 bash[18607]: audit 2024-04-25T19:53:12.821244+0000 mon.a (mon.0) 58 : audit [INF] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' 2024-04-25T19:53:13.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:13 smithi132 bash[18607]: audit 2024-04-25T19:53:12.867333+0000 mon.a (mon.0) 59 : audit [DBG] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:53:13.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:13 smithi132 bash[18607]: audit 2024-04-25T19:53:12.867333+0000 mon.a (mon.0) 59 : audit [DBG] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:53:14.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:14 smithi132 bash[18607]: audit 2024-04-25T19:53:12.812356+0000 mgr.x (mgr.14120) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:53:14.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:14 smithi132 bash[18607]: audit 2024-04-25T19:53:12.812356+0000 mgr.x (mgr.14120) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:53:15.223 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-25T19:53:15.224 INFO:teuthology.orchestra.run.smithi132.stdout:Generating ssh key... 2024-04-25T19:53:16.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:15 smithi132 bash[18607]: audit 2024-04-25T19:53:14.700112+0000 mgr.x (mgr.14120) 9 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:53:16.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:15 smithi132 bash[18607]: audit 2024-04-25T19:53:14.700112+0000 mgr.x (mgr.14120) 9 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:53:17.001 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:16 smithi132 bash[18607]: audit 2024-04-25T19:53:16.436422+0000 mgr.x (mgr.14120) 10 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:53:17.002 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:16 smithi132 bash[18607]: audit 2024-04-25T19:53:16.436422+0000 mgr.x (mgr.14120) 10 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:53:17.002 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:16 smithi132 bash[18607]: cephadm 2024-04-25T19:53:16.436888+0000 mgr.x (mgr.14120) 11 : cephadm [INF] Generating ssh key... 2024-04-25T19:53:17.002 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:16 smithi132 bash[18607]: cephadm 2024-04-25T19:53:16.436888+0000 mgr.x (mgr.14120) 11 : cephadm [INF] Generating ssh key... 2024-04-25T19:53:17.002 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:16 smithi132 bash[18860]: Generating public/private rsa key pair. 2024-04-25T19:53:17.002 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:16 smithi132 bash[18860]: Your identification has been saved in /tmp/tmpe35r63jy/key. 2024-04-25T19:53:17.002 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:16 smithi132 bash[18860]: Your public key has been saved in /tmp/tmpe35r63jy/key.pub. 2024-04-25T19:53:17.002 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:16 smithi132 bash[18860]: The key fingerprint is: 2024-04-25T19:53:17.002 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:16 smithi132 bash[18860]: SHA256:G3IVBNqU/WvCZXMGKDlE2ji4JU9XjuE/j9RikjKJtk0 ceph-3114c848-033d-11ef-bc93-c7b262605968 2024-04-25T19:53:17.002 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:16 smithi132 bash[18860]: The key's randomart image is: 2024-04-25T19:53:17.002 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:16 smithi132 bash[18860]: +---[RSA 3072]----+ 2024-04-25T19:53:17.003 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:16 smithi132 bash[18860]: | o**+. | 2024-04-25T19:53:17.003 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:16 smithi132 bash[18860]: | . O==o.. | 2024-04-25T19:53:17.003 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:16 smithi132 bash[18860]: | o B *oo. . | 2024-04-25T19:53:17.003 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:16 smithi132 bash[18860]: | B + + .= o | 2024-04-25T19:53:17.003 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:16 smithi132 bash[18860]: | + E S.*o.= | 2024-04-25T19:53:17.003 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:16 smithi132 bash[18860]: | . + = *o=o | 2024-04-25T19:53:17.003 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:16 smithi132 bash[18860]: | . . . .o. | 2024-04-25T19:53:17.003 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:16 smithi132 bash[18860]: | | 2024-04-25T19:53:17.003 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:16 smithi132 bash[18860]: | | 2024-04-25T19:53:17.003 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:16 smithi132 bash[18860]: +----[SHA256]-----+ 2024-04-25T19:53:18.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:17 smithi132 bash[18607]: audit 2024-04-25T19:53:16.811887+0000 mon.a (mon.0) 60 : audit [INF] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' 2024-04-25T19:53:18.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:17 smithi132 bash[18607]: audit 2024-04-25T19:53:16.811887+0000 mon.a (mon.0) 60 : audit [INF] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' 2024-04-25T19:53:18.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:17 smithi132 bash[18607]: audit 2024-04-25T19:53:16.816802+0000 mon.a (mon.0) 61 : audit [INF] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' 2024-04-25T19:53:18.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:17 smithi132 bash[18607]: audit 2024-04-25T19:53:16.816802+0000 mon.a (mon.0) 61 : audit [INF] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' 2024-04-25T19:53:19.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:18 smithi132 bash[18607]: audit 2024-04-25T19:53:18.573500+0000 mgr.x (mgr.14120) 12 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:53:19.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:18 smithi132 bash[18607]: audit 2024-04-25T19:53:18.573500+0000 mgr.x (mgr.14120) 12 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:53:19.133 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDfCvz/NW7eUGZIYB4V3Ysfe/+UnWGxEg7wcFyp+m79OnMyLj/nJCRBTz7nGI7BRzmp6w5hZGp3w8/uepc5uQ4NmInZKR4jQ4yU+LvEISOmmwNM05/eOAm2PBQDdtg06dEdKTrXKhYmcC+Dco66rHA9bKs/UsEqgRMr7p+MisgzE4oR9EJwqCowQL5Atxqsgcifr0800xSpPTXCKmYBH6eyn+/JcaJ1wl8hwIpeajOFsrH/WObTdKnILdj6p9GnHdy80oJ+c9vgSSzyfeKe3tUluL4cj3fo+l8Pc3NmDBDb61DrlUdqMdHuKQphvCvH5bBI/50dTBb3OaQ/gmOmApYjSC9Nx0t8H7vMPA+6VuhwacqEgcycNrsW2Bb7HLBHwGXW7oTizXNk0Pd3qBekXXNTgx/HPkqUApvxwz2N5uS5QaEJUgaD3M3NZz7HpBZ16VvE/3QPTwwXl/VYnlAhE1HgHMX77pA5GUIoTozDDQj9IzXLzeTFlewyRrCfzWVgFi0= ceph-3114c848-033d-11ef-bc93-c7b262605968 2024-04-25T19:53:19.133 INFO:teuthology.orchestra.run.smithi132.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-25T19:53:19.133 INFO:teuthology.orchestra.run.smithi132.stdout:Adding key to root@localhost authorized_keys... 2024-04-25T19:53:19.134 INFO:teuthology.orchestra.run.smithi132.stdout:Adding host smithi132... 2024-04-25T19:53:20.762 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:20 smithi132 bash[18607]: audit 2024-04-25T19:53:20.486805+0000 mgr.x (mgr.14120) 13 : audit [DBG] from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi132", "addr": "172.21.15.132", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:53:20.763 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:20 smithi132 bash[18607]: audit 2024-04-25T19:53:20.486805+0000 mgr.x (mgr.14120) 13 : audit [DBG] from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi132", "addr": "172.21.15.132", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:53:22.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:21 smithi132 bash[18607]: cephadm 2024-04-25T19:53:21.027865+0000 mgr.x (mgr.14120) 14 : cephadm [INF] Deploying cephadm binary to smithi132 2024-04-25T19:53:22.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:21 smithi132 bash[18607]: cephadm 2024-04-25T19:53:21.027865+0000 mgr.x (mgr.14120) 14 : cephadm [INF] Deploying cephadm binary to smithi132 2024-04-25T19:53:26.693 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout Added host 'smithi132' with addr '172.21.15.132' 2024-04-25T19:53:26.693 INFO:teuthology.orchestra.run.smithi132.stdout:Deploying unmanaged mon service... 2024-04-25T19:53:27.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:26 smithi132 bash[18607]: audit 2024-04-25T19:53:25.906420+0000 mon.a (mon.0) 62 : audit [INF] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' 2024-04-25T19:53:27.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:26 smithi132 bash[18607]: audit 2024-04-25T19:53:25.906420+0000 mon.a (mon.0) 62 : audit [INF] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' 2024-04-25T19:53:27.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:26 smithi132 bash[18607]: cephadm 2024-04-25T19:53:25.907107+0000 mgr.x (mgr.14120) 15 : cephadm [INF] Added host smithi132 2024-04-25T19:53:27.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:26 smithi132 bash[18607]: cephadm 2024-04-25T19:53:25.907107+0000 mgr.x (mgr.14120) 15 : cephadm [INF] Added host smithi132 2024-04-25T19:53:27.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:26 smithi132 bash[18607]: audit 2024-04-25T19:53:25.948679+0000 mon.a (mon.0) 63 : audit [DBG] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:53:27.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:26 smithi132 bash[18607]: audit 2024-04-25T19:53:25.948679+0000 mon.a (mon.0) 63 : audit [DBG] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:53:29.156 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-25T19:53:29.156 INFO:teuthology.orchestra.run.smithi132.stdout:Deploying unmanaged mgr service... 2024-04-25T19:53:29.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:29 smithi132 bash[18607]: audit 2024-04-25T19:53:28.401407+0000 mgr.x (mgr.14120) 16 : audit [DBG] from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:53:29.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:29 smithi132 bash[18607]: audit 2024-04-25T19:53:28.401407+0000 mgr.x (mgr.14120) 16 : audit [DBG] from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:53:29.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:29 smithi132 bash[18607]: cephadm 2024-04-25T19:53:28.404031+0000 mgr.x (mgr.14120) 17 : cephadm [INF] Saving service mon spec with placement count:5 2024-04-25T19:53:29.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:29 smithi132 bash[18607]: cephadm 2024-04-25T19:53:28.404031+0000 mgr.x (mgr.14120) 17 : cephadm [INF] Saving service mon spec with placement count:5 2024-04-25T19:53:29.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:29 smithi132 bash[18607]: audit 2024-04-25T19:53:28.406515+0000 mon.a (mon.0) 64 : audit [INF] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' 2024-04-25T19:53:29.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:29 smithi132 bash[18607]: audit 2024-04-25T19:53:28.406515+0000 mon.a (mon.0) 64 : audit [INF] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' 2024-04-25T19:53:29.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:29 smithi132 bash[18607]: audit 2024-04-25T19:53:28.925702+0000 mon.a (mon.0) 65 : audit [INF] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' 2024-04-25T19:53:29.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:29 smithi132 bash[18607]: audit 2024-04-25T19:53:28.925702+0000 mon.a (mon.0) 65 : audit [INF] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' 2024-04-25T19:53:30.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:30 smithi132 bash[18607]: audit 2024-04-25T19:53:29.464246+0000 mon.a (mon.0) 66 : audit [INF] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' 2024-04-25T19:53:30.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:30 smithi132 bash[18607]: audit 2024-04-25T19:53:29.464246+0000 mon.a (mon.0) 66 : audit [INF] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' 2024-04-25T19:53:30.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:30 smithi132 bash[18607]: audit 2024-04-25T19:53:30.428044+0000 mon.a (mon.0) 67 : audit [INF] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' 2024-04-25T19:53:30.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:30 smithi132 bash[18607]: audit 2024-04-25T19:53:30.428044+0000 mon.a (mon.0) 67 : audit [INF] from='mgr.14120 172.21.15.132:0/4244208503' entity='mgr.x' 2024-04-25T19:53:31.073 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-25T19:53:31.724 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:31 smithi132 bash[18607]: cluster 2024-04-25T19:53:29.966474+0000 mgr.x (mgr.14120) 18 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:53:31.724 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:31 smithi132 bash[18607]: cluster 2024-04-25T19:53:29.966474+0000 mgr.x (mgr.14120) 18 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:53:31.724 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:31 smithi132 bash[18607]: audit 2024-04-25T19:53:30.420625+0000 mgr.x (mgr.14120) 19 : audit [DBG] from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:53:31.724 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:31 smithi132 bash[18607]: audit 2024-04-25T19:53:30.420625+0000 mgr.x (mgr.14120) 19 : audit [DBG] from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:53:31.724 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:31 smithi132 bash[18607]: cephadm 2024-04-25T19:53:30.423260+0000 mgr.x (mgr.14120) 20 : cephadm [INF] Saving service mgr spec with placement count:2 2024-04-25T19:53:31.724 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:31 smithi132 bash[18607]: cephadm 2024-04-25T19:53:30.423260+0000 mgr.x (mgr.14120) 20 : cephadm [INF] Saving service mgr spec with placement count:2 2024-04-25T19:53:33.606 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:33 smithi132 bash[18607]: cluster 2024-04-25T19:53:31.966753+0000 mgr.x (mgr.14120) 21 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:53:33.606 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:33 smithi132 bash[18607]: cluster 2024-04-25T19:53:31.966753+0000 mgr.x (mgr.14120) 21 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:53:33.607 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:33 smithi132 bash[18607]: audit 2024-04-25T19:53:32.327854+0000 mon.a (mon.0) 68 : audit [INF] from='client.? 172.21.15.132:0/3737308754' entity='client.admin' 2024-04-25T19:53:33.607 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:33 smithi132 bash[18607]: audit 2024-04-25T19:53:32.327854+0000 mon.a (mon.0) 68 : audit [INF] from='client.? 172.21.15.132:0/3737308754' entity='client.admin' 2024-04-25T19:53:34.716 INFO:teuthology.orchestra.run.smithi132.stdout:Enabling the dashboard module... 2024-04-25T19:53:35.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:35 smithi132 bash[18607]: cluster 2024-04-25T19:53:33.967121+0000 mgr.x (mgr.14120) 22 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:53:35.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:35 smithi132 bash[18607]: cluster 2024-04-25T19:53:33.967121+0000 mgr.x (mgr.14120) 22 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:53:35.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:35 smithi132 bash[18607]: audit 2024-04-25T19:53:34.217964+0000 mon.a (mon.0) 69 : audit [INF] from='client.? 172.21.15.132:0/2916620112' entity='client.admin' 2024-04-25T19:53:35.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:35 smithi132 bash[18607]: audit 2024-04-25T19:53:34.217964+0000 mon.a (mon.0) 69 : audit [INF] from='client.? 172.21.15.132:0/2916620112' entity='client.admin' 2024-04-25T19:53:36.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:36 smithi132 bash[18607]: audit 2024-04-25T19:53:36.072507+0000 mon.a (mon.0) 70 : audit [INF] from='client.? 172.21.15.132:0/3577329272' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-25T19:53:36.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:36 smithi132 bash[18607]: audit 2024-04-25T19:53:36.072507+0000 mon.a (mon.0) 70 : audit [INF] from='client.? 172.21.15.132:0/3577329272' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-25T19:53:36.521 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:36 smithi132 bash[18860]: ignoring --setuser ceph since I am not root 2024-04-25T19:53:36.521 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:36 smithi132 bash[18860]: ignoring --setgroup ceph since I am not root 2024-04-25T19:53:36.521 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:36 smithi132 bash[18860]: debug 2024-04-25T19:53:36.412+0000 7f7730c8a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T19:53:37.494 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:37 smithi132 bash[18607]: audit 2024-04-25T19:53:36.222909+0000 mon.a (mon.0) 71 : audit [INF] from='client.? 172.21.15.132:0/3577329272' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-25T19:53:37.494 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:37 smithi132 bash[18607]: audit 2024-04-25T19:53:36.222909+0000 mon.a (mon.0) 71 : audit [INF] from='client.? 172.21.15.132:0/3577329272' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-25T19:53:37.494 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:37 smithi132 bash[18607]: cluster 2024-04-25T19:53:36.229442+0000 mon.a (mon.0) 72 : cluster [DBG] mgrmap e9: x(active, since 26s) 2024-04-25T19:53:37.494 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:37 smithi132 bash[18607]: cluster 2024-04-25T19:53:36.229442+0000 mon.a (mon.0) 72 : cluster [DBG] mgrmap e9: x(active, since 26s) 2024-04-25T19:53:38.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:38 smithi132 bash[18607]: audit 2024-04-25T19:53:38.181199+0000 mon.a (mon.0) 73 : audit [DBG] from='client.? 172.21.15.132:0/233594630' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-25T19:53:38.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:38 smithi132 bash[18607]: audit 2024-04-25T19:53:38.181199+0000 mon.a (mon.0) 73 : audit [DBG] from='client.? 172.21.15.132:0/233594630' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-25T19:53:38.271 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:38 smithi132 bash[18860]: debug 2024-04-25T19:53:38.012+0000 7f7730c8a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T19:53:38.271 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:38 smithi132 bash[18860]: debug 2024-04-25T19:53:38.112+0000 7f7730c8a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T19:53:38.771 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:38 smithi132 bash[18860]: debug 2024-04-25T19:53:38.364+0000 7f7730c8a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T19:53:38.836 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout { 2024-04-25T19:53:38.836 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-25T19:53:38.836 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T19:53:38.836 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-25T19:53:38.836 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-25T19:53:38.836 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout } 2024-04-25T19:53:38.836 INFO:teuthology.orchestra.run.smithi132.stdout:Waiting for the mgr to restart... 2024-04-25T19:53:38.836 INFO:teuthology.orchestra.run.smithi132.stdout:Waiting for mgr epoch 9... 2024-04-25T19:53:39.437 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:39 smithi132 bash[18860]: debug 2024-04-25T19:53:39.168+0000 7f7730c8a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T19:53:39.437 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:39 smithi132 bash[18860]: debug 2024-04-25T19:53:39.280+0000 7f7730c8a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T19:53:39.437 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:39 smithi132 bash[18860]: debug 2024-04-25T19:53:39.384+0000 7f7730c8a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T19:53:40.021 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:39 smithi132 bash[18860]: debug 2024-04-25T19:53:39.748+0000 7f7730c8a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T19:53:40.021 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:39 smithi132 bash[18860]: debug 2024-04-25T19:53:39.864+0000 7f7730c8a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T19:53:40.021 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:39 smithi132 bash[18860]: debug 2024-04-25T19:53:39.968+0000 7f7730c8a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T19:53:40.771 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:40 smithi132 bash[18860]: debug 2024-04-25T19:53:40.328+0000 7f7730c8a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T19:53:40.771 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:40 smithi132 bash[18860]: debug 2024-04-25T19:53:40.568+0000 7f7730c8a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T19:53:41.215 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:40 smithi132 bash[18860]: debug 2024-04-25T19:53:40.828+0000 7f7730c8a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T19:53:41.215 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:40 smithi132 bash[18860]: debug 2024-04-25T19:53:40.944+0000 7f7730c8a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T19:53:41.215 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:41 smithi132 bash[18860]: debug 2024-04-25T19:53:41.216+0000 7f7730c8a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T19:53:41.989 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:41 smithi132 bash[18860]: debug 2024-04-25T19:53:41.632+0000 7f7730c8a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T19:53:41.989 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:41 smithi132 bash[18860]: debug 2024-04-25T19:53:41.872+0000 7f7730c8a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T19:53:42.521 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:41 smithi132 bash[18860]: debug 2024-04-25T19:53:41.992+0000 7f7730c8a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T19:53:42.521 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:42 smithi132 bash[18860]: debug 2024-04-25T19:53:42.244+0000 7f7730c8a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T19:53:43.521 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:43 smithi132 bash[18860]: debug 2024-04-25T19:53:43.263+0000 7f7730c8a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T19:53:43.521 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:43 smithi132 bash[18860]: debug 2024-04-25T19:53:43.379+0000 7f7730c8a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T19:53:44.021 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:43 smithi132 bash[18860]: debug 2024-04-25T19:53:43.679+0000 7f7730c8a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T19:53:44.021 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:43 smithi132 bash[18860]: debug 2024-04-25T19:53:43.795+0000 7f7730c8a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T19:53:44.521 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:44 smithi132 bash[18860]: debug 2024-04-25T19:53:44.079+0000 7f7730c8a200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-04-25T19:53:44.787 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:44 smithi132 bash[18860]: debug 2024-04-25T19:53:44.531+0000 7f7730c8a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T19:53:44.787 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:53:44 smithi132 bash[18860]: debug 2024-04-25T19:53:44.655+0000 7f7730c8a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T19:53:44.787 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:44 smithi132 bash[18607]: cluster 2024-04-25T19:53:44.678468+0000 mon.a (mon.0) 74 : cluster [INF] Active manager daemon x restarted 2024-04-25T19:53:44.788 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:44 smithi132 bash[18607]: cluster 2024-04-25T19:53:44.678468+0000 mon.a (mon.0) 74 : cluster [INF] Active manager daemon x restarted 2024-04-25T19:53:44.788 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:44 smithi132 bash[18607]: cluster 2024-04-25T19:53:44.678961+0000 mon.a (mon.0) 75 : cluster [INF] Activating manager daemon x 2024-04-25T19:53:44.788 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:44 smithi132 bash[18607]: cluster 2024-04-25T19:53:44.678961+0000 mon.a (mon.0) 75 : cluster [INF] Activating manager daemon x 2024-04-25T19:53:44.788 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:44 smithi132 bash[18607]: cluster 2024-04-25T19:53:44.686650+0000 mon.a (mon.0) 76 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-04-25T19:53:44.788 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:44 smithi132 bash[18607]: cluster 2024-04-25T19:53:44.686650+0000 mon.a (mon.0) 76 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-04-25T19:53:44.788 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:44 smithi132 bash[18607]: cluster 2024-04-25T19:53:44.686903+0000 mon.a (mon.0) 77 : cluster [DBG] mgrmap e10: x(active, starting, since 0.00813564s) 2024-04-25T19:53:44.788 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:44 smithi132 bash[18607]: cluster 2024-04-25T19:53:44.686903+0000 mon.a (mon.0) 77 : cluster [DBG] mgrmap e10: x(active, starting, since 0.00813564s) 2024-04-25T19:53:44.788 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:44 smithi132 bash[18607]: audit 2024-04-25T19:53:44.688027+0000 mon.a (mon.0) 78 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T19:53:44.788 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:44 smithi132 bash[18607]: audit 2024-04-25T19:53:44.688027+0000 mon.a (mon.0) 78 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T19:53:44.788 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:44 smithi132 bash[18607]: audit 2024-04-25T19:53:44.689854+0000 mon.a (mon.0) 79 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T19:53:44.788 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:44 smithi132 bash[18607]: audit 2024-04-25T19:53:44.689854+0000 mon.a (mon.0) 79 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T19:53:44.788 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:44 smithi132 bash[18607]: audit 2024-04-25T19:53:44.690959+0000 mon.a (mon.0) 80 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T19:53:44.789 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:44 smithi132 bash[18607]: audit 2024-04-25T19:53:44.690959+0000 mon.a (mon.0) 80 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T19:53:44.789 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:44 smithi132 bash[18607]: audit 2024-04-25T19:53:44.691207+0000 mon.a (mon.0) 81 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T19:53:44.789 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:44 smithi132 bash[18607]: audit 2024-04-25T19:53:44.691207+0000 mon.a (mon.0) 81 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T19:53:44.789 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:44 smithi132 bash[18607]: audit 2024-04-25T19:53:44.691379+0000 mon.a (mon.0) 82 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T19:53:44.789 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:44 smithi132 bash[18607]: audit 2024-04-25T19:53:44.691379+0000 mon.a (mon.0) 82 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T19:53:44.789 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:44 smithi132 bash[18607]: cluster 2024-04-25T19:53:44.718849+0000 mon.a (mon.0) 83 : cluster [INF] Manager daemon x is now available 2024-04-25T19:53:44.789 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:44 smithi132 bash[18607]: cluster 2024-04-25T19:53:44.718849+0000 mon.a (mon.0) 83 : cluster [INF] Manager daemon x is now available 2024-04-25T19:53:46.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:45 smithi132 bash[18607]: audit 2024-04-25T19:53:44.744008+0000 mon.a (mon.0) 84 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:53:46.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:45 smithi132 bash[18607]: audit 2024-04-25T19:53:44.744008+0000 mon.a (mon.0) 84 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:53:46.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:45 smithi132 bash[18607]: audit 2024-04-25T19:53:44.763844+0000 mon.a (mon.0) 85 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T19:53:46.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:45 smithi132 bash[18607]: audit 2024-04-25T19:53:44.763844+0000 mon.a (mon.0) 85 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T19:53:46.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:45 smithi132 bash[18607]: audit 2024-04-25T19:53:44.767244+0000 mon.a (mon.0) 86 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T19:53:46.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:45 smithi132 bash[18607]: audit 2024-04-25T19:53:44.767244+0000 mon.a (mon.0) 86 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T19:53:46.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:45 smithi132 bash[18607]: cephadm 2024-04-25T19:53:45.105157+0000 mgr.x (mgr.14152) 1 : cephadm [INF] [25/Apr/2024:19:53:45] ENGINE Bus STARTING 2024-04-25T19:53:46.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:45 smithi132 bash[18607]: cephadm 2024-04-25T19:53:45.105157+0000 mgr.x (mgr.14152) 1 : cephadm [INF] [25/Apr/2024:19:53:45] ENGINE Bus STARTING 2024-04-25T19:53:46.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:45 smithi132 bash[18607]: cephadm 2024-04-25T19:53:45.223358+0000 mgr.x (mgr.14152) 2 : cephadm [INF] [25/Apr/2024:19:53:45] ENGINE Serving on https://172.21.15.132:7150 2024-04-25T19:53:46.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:45 smithi132 bash[18607]: cephadm 2024-04-25T19:53:45.223358+0000 mgr.x (mgr.14152) 2 : cephadm [INF] [25/Apr/2024:19:53:45] ENGINE Serving on https://172.21.15.132:7150 2024-04-25T19:53:46.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:45 smithi132 bash[18607]: cluster 2024-04-25T19:53:45.691424+0000 mon.a (mon.0) 87 : cluster [DBG] mgrmap e11: x(active, since 1.01266s) 2024-04-25T19:53:46.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:45 smithi132 bash[18607]: cluster 2024-04-25T19:53:45.691424+0000 mon.a (mon.0) 87 : cluster [DBG] mgrmap e11: x(active, since 1.01266s) 2024-04-25T19:53:46.325 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout { 2024-04-25T19:53:46.325 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-25T19:53:46.325 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-25T19:53:46.325 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout } 2024-04-25T19:53:46.325 INFO:teuthology.orchestra.run.smithi132.stdout:mgr epoch 9 is available 2024-04-25T19:53:46.325 INFO:teuthology.orchestra.run.smithi132.stdout:Generating a dashboard self-signed certificate... 2024-04-25T19:53:46.987 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:46 smithi132 bash[18607]: cephadm 2024-04-25T19:53:45.324803+0000 mgr.x (mgr.14152) 3 : cephadm [INF] [25/Apr/2024:19:53:45] ENGINE Serving on http://172.21.15.132:8765 2024-04-25T19:53:46.987 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:46 smithi132 bash[18607]: cephadm 2024-04-25T19:53:45.324803+0000 mgr.x (mgr.14152) 3 : cephadm [INF] [25/Apr/2024:19:53:45] ENGINE Serving on http://172.21.15.132:8765 2024-04-25T19:53:46.987 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:46 smithi132 bash[18607]: cephadm 2024-04-25T19:53:45.324955+0000 mgr.x (mgr.14152) 4 : cephadm [INF] [25/Apr/2024:19:53:45] ENGINE Bus STARTED 2024-04-25T19:53:46.987 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:46 smithi132 bash[18607]: cephadm 2024-04-25T19:53:45.324955+0000 mgr.x (mgr.14152) 4 : cephadm [INF] [25/Apr/2024:19:53:45] ENGINE Bus STARTED 2024-04-25T19:53:46.988 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:46 smithi132 bash[18607]: audit 2024-04-25T19:53:45.690873+0000 mgr.x (mgr.14152) 5 : audit [DBG] from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-25T19:53:46.988 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:46 smithi132 bash[18607]: audit 2024-04-25T19:53:45.690873+0000 mgr.x (mgr.14152) 5 : audit [DBG] from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-25T19:53:46.988 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:46 smithi132 bash[18607]: audit 2024-04-25T19:53:45.708274+0000 mgr.x (mgr.14152) 6 : audit [DBG] from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-25T19:53:46.988 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:46 smithi132 bash[18607]: audit 2024-04-25T19:53:45.708274+0000 mgr.x (mgr.14152) 6 : audit [DBG] from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-25T19:53:48.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:47 smithi132 bash[18607]: cluster 2024-04-25T19:53:46.744800+0000 mon.a (mon.0) 88 : cluster [DBG] mgrmap e12: x(active, since 2s) 2024-04-25T19:53:48.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:47 smithi132 bash[18607]: cluster 2024-04-25T19:53:46.744800+0000 mon.a (mon.0) 88 : cluster [DBG] mgrmap e12: x(active, since 2s) 2024-04-25T19:53:48.288 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-25T19:53:48.288 INFO:teuthology.orchestra.run.smithi132.stdout:Creating initial admin user... 2024-04-25T19:53:49.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:48 smithi132 bash[18607]: audit 2024-04-25T19:53:47.575808+0000 mgr.x (mgr.14152) 7 : audit [DBG] from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:53:49.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:48 smithi132 bash[18607]: audit 2024-04-25T19:53:47.575808+0000 mgr.x (mgr.14152) 7 : audit [DBG] from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:53:49.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:48 smithi132 bash[18607]: audit 2024-04-25T19:53:47.767951+0000 mon.a (mon.0) 89 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:53:49.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:48 smithi132 bash[18607]: audit 2024-04-25T19:53:47.767951+0000 mon.a (mon.0) 89 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:53:49.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:48 smithi132 bash[18607]: audit 2024-04-25T19:53:47.770731+0000 mon.a (mon.0) 90 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:53:49.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:48 smithi132 bash[18607]: audit 2024-04-25T19:53:47.770731+0000 mon.a (mon.0) 90 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:53:50.331 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$P/azijDNysdjHuu/M.6EweCi/oQQuB.1Sk7m72tvnhT326Ulf/lbC", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714074829, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-25T19:53:50.331 INFO:teuthology.orchestra.run.smithi132.stdout:Fetching dashboard port number... 2024-04-25T19:53:51.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:50 smithi132 bash[18607]: audit 2024-04-25T19:53:49.527008+0000 mgr.x (mgr.14152) 8 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:53:51.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:50 smithi132 bash[18607]: audit 2024-04-25T19:53:49.527008+0000 mgr.x (mgr.14152) 8 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:53:51.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:50 smithi132 bash[18607]: audit 2024-04-25T19:53:49.763377+0000 mon.a (mon.0) 91 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:53:51.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:50 smithi132 bash[18607]: audit 2024-04-25T19:53:49.763377+0000 mon.a (mon.0) 91 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:53:51.972 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:51 smithi132 bash[18607]: cluster 2024-04-25T19:53:50.770031+0000 mon.a (mon.0) 92 : cluster [DBG] mgrmap e13: x(active, since 6s) 2024-04-25T19:53:51.972 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:51 smithi132 bash[18607]: cluster 2024-04-25T19:53:50.770031+0000 mon.a (mon.0) 92 : cluster [DBG] mgrmap e13: x(active, since 6s) 2024-04-25T19:53:51.972 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:51 smithi132 bash[18607]: audit 2024-04-25T19:53:51.519907+0000 mon.a (mon.0) 93 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:53:51.972 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:51 smithi132 bash[18607]: audit 2024-04-25T19:53:51.519907+0000 mon.a (mon.0) 93 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:53:51.973 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:51 smithi132 bash[18607]: audit 2024-04-25T19:53:51.653237+0000 mon.a (mon.0) 94 : audit [DBG] from='client.? 172.21.15.132:0/3947812433' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-25T19:53:51.973 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:51 smithi132 bash[18607]: audit 2024-04-25T19:53:51.653237+0000 mon.a (mon.0) 94 : audit [DBG] from='client.? 172.21.15.132:0/3947812433' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-25T19:53:52.248 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stdout 8443 2024-04-25T19:53:52.248 INFO:teuthology.orchestra.run.smithi132.stdout:firewalld does not appear to be present 2024-04-25T19:53:52.248 INFO:teuthology.orchestra.run.smithi132.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-25T19:53:52.250 INFO:teuthology.orchestra.run.smithi132.stdout:Ceph Dashboard is now available at: 2024-04-25T19:53:52.250 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T19:53:52.250 INFO:teuthology.orchestra.run.smithi132.stdout: URL: https://smithi132.front.sepia.ceph.com:8443/ 2024-04-25T19:53:52.251 INFO:teuthology.orchestra.run.smithi132.stdout: User: admin 2024-04-25T19:53:52.251 INFO:teuthology.orchestra.run.smithi132.stdout: Password: 67qtx14rml 2024-04-25T19:53:52.251 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T19:53:52.251 INFO:teuthology.orchestra.run.smithi132.stdout:Saving cluster configuration to /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/config directory 2024-04-25T19:53:52.251 INFO:teuthology.orchestra.run.smithi132.stdout:Enabling autotune for osd_memory_target 2024-04-25T19:53:54.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:53 smithi132 bash[18607]: audit 2024-04-25T19:53:52.710928+0000 mon.a (mon.0) 95 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:53:54.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:53 smithi132 bash[18607]: audit 2024-04-25T19:53:52.710928+0000 mon.a (mon.0) 95 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:53:56.160 INFO:teuthology.orchestra.run.smithi132.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-25T19:53:56.160 INFO:teuthology.orchestra.run.smithi132.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-25T19:53:56.160 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T19:53:56.160 INFO:teuthology.orchestra.run.smithi132.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 3114c848-033d-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-25T19:53:56.160 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T19:53:56.160 INFO:teuthology.orchestra.run.smithi132.stdout:Or, if you are only running a single cluster on this host: 2024-04-25T19:53:56.161 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T19:53:56.161 INFO:teuthology.orchestra.run.smithi132.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-25T19:53:56.161 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T19:53:56.161 INFO:teuthology.orchestra.run.smithi132.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-25T19:53:56.161 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T19:53:56.161 INFO:teuthology.orchestra.run.smithi132.stdout: ceph telemetry on 2024-04-25T19:53:56.161 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T19:53:56.161 INFO:teuthology.orchestra.run.smithi132.stdout:For more information see: 2024-04-25T19:53:56.161 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T19:53:56.161 INFO:teuthology.orchestra.run.smithi132.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-25T19:53:56.161 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T19:53:56.161 INFO:teuthology.orchestra.run.smithi132.stdout:Bootstrap complete. 2024-04-25T19:53:56.219 INFO:tasks.cephadm:Fetching config... 2024-04-25T19:53:56.219 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-25T19:53:56.220 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-25T19:53:56.225 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-25T19:53:56.225 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-25T19:53:56.225 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-25T19:53:56.274 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-25T19:53:56.275 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-25T19:53:56.275 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-25T19:53:56.331 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-25T19:53:56.331 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-25T19:53:56.331 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-25T19:53:56.379 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-25T19:53:56.379 DEBUG:teuthology.orchestra.run.smithi132:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDfCvz/NW7eUGZIYB4V3Ysfe/+UnWGxEg7wcFyp+m79OnMyLj/nJCRBTz7nGI7BRzmp6w5hZGp3w8/uepc5uQ4NmInZKR4jQ4yU+LvEISOmmwNM05/eOAm2PBQDdtg06dEdKTrXKhYmcC+Dco66rHA9bKs/UsEqgRMr7p+MisgzE4oR9EJwqCowQL5Atxqsgcifr0800xSpPTXCKmYBH6eyn+/JcaJ1wl8hwIpeajOFsrH/WObTdKnILdj6p9GnHdy80oJ+c9vgSSzyfeKe3tUluL4cj3fo+l8Pc3NmDBDb61DrlUdqMdHuKQphvCvH5bBI/50dTBb3OaQ/gmOmApYjSC9Nx0t8H7vMPA+6VuhwacqEgcycNrsW2Bb7HLBHwGXW7oTizXNk0Pd3qBekXXNTgx/HPkqUApvxwz2N5uS5QaEJUgaD3M3NZz7HpBZ16VvE/3QPTwwXl/VYnlAhE1HgHMX77pA5GUIoTozDDQj9IzXLzeTFlewyRrCfzWVgFi0= ceph-3114c848-033d-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-25T19:53:56.442 INFO:teuthology.orchestra.run.smithi132.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDfCvz/NW7eUGZIYB4V3Ysfe/+UnWGxEg7wcFyp+m79OnMyLj/nJCRBTz7nGI7BRzmp6w5hZGp3w8/uepc5uQ4NmInZKR4jQ4yU+LvEISOmmwNM05/eOAm2PBQDdtg06dEdKTrXKhYmcC+Dco66rHA9bKs/UsEqgRMr7p+MisgzE4oR9EJwqCowQL5Atxqsgcifr0800xSpPTXCKmYBH6eyn+/JcaJ1wl8hwIpeajOFsrH/WObTdKnILdj6p9GnHdy80oJ+c9vgSSzyfeKe3tUluL4cj3fo+l8Pc3NmDBDb61DrlUdqMdHuKQphvCvH5bBI/50dTBb3OaQ/gmOmApYjSC9Nx0t8H7vMPA+6VuhwacqEgcycNrsW2Bb7HLBHwGXW7oTizXNk0Pd3qBekXXNTgx/HPkqUApvxwz2N5uS5QaEJUgaD3M3NZz7HpBZ16VvE/3QPTwwXl/VYnlAhE1HgHMX77pA5GUIoTozDDQj9IzXLzeTFlewyRrCfzWVgFi0= ceph-3114c848-033d-11ef-bc93-c7b262605968 2024-04-25T19:53:56.451 DEBUG:teuthology.orchestra.run.smithi150:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDfCvz/NW7eUGZIYB4V3Ysfe/+UnWGxEg7wcFyp+m79OnMyLj/nJCRBTz7nGI7BRzmp6w5hZGp3w8/uepc5uQ4NmInZKR4jQ4yU+LvEISOmmwNM05/eOAm2PBQDdtg06dEdKTrXKhYmcC+Dco66rHA9bKs/UsEqgRMr7p+MisgzE4oR9EJwqCowQL5Atxqsgcifr0800xSpPTXCKmYBH6eyn+/JcaJ1wl8hwIpeajOFsrH/WObTdKnILdj6p9GnHdy80oJ+c9vgSSzyfeKe3tUluL4cj3fo+l8Pc3NmDBDb61DrlUdqMdHuKQphvCvH5bBI/50dTBb3OaQ/gmOmApYjSC9Nx0t8H7vMPA+6VuhwacqEgcycNrsW2Bb7HLBHwGXW7oTizXNk0Pd3qBekXXNTgx/HPkqUApvxwz2N5uS5QaEJUgaD3M3NZz7HpBZ16VvE/3QPTwwXl/VYnlAhE1HgHMX77pA5GUIoTozDDQj9IzXLzeTFlewyRrCfzWVgFi0= ceph-3114c848-033d-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-25T19:53:56.475 INFO:teuthology.orchestra.run.smithi150.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDfCvz/NW7eUGZIYB4V3Ysfe/+UnWGxEg7wcFyp+m79OnMyLj/nJCRBTz7nGI7BRzmp6w5hZGp3w8/uepc5uQ4NmInZKR4jQ4yU+LvEISOmmwNM05/eOAm2PBQDdtg06dEdKTrXKhYmcC+Dco66rHA9bKs/UsEqgRMr7p+MisgzE4oR9EJwqCowQL5Atxqsgcifr0800xSpPTXCKmYBH6eyn+/JcaJ1wl8hwIpeajOFsrH/WObTdKnILdj6p9GnHdy80oJ+c9vgSSzyfeKe3tUluL4cj3fo+l8Pc3NmDBDb61DrlUdqMdHuKQphvCvH5bBI/50dTBb3OaQ/gmOmApYjSC9Nx0t8H7vMPA+6VuhwacqEgcycNrsW2Bb7HLBHwGXW7oTizXNk0Pd3qBekXXNTgx/HPkqUApvxwz2N5uS5QaEJUgaD3M3NZz7HpBZ16VvE/3QPTwwXl/VYnlAhE1HgHMX77pA5GUIoTozDDQj9IzXLzeTFlewyRrCfzWVgFi0= ceph-3114c848-033d-11ef-bc93-c7b262605968 2024-04-25T19:53:56.484 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-25T19:53:56.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:56 smithi132 bash[18607]: audit 2024-04-25T19:53:55.618990+0000 mon.a (mon.0) 96 : audit [INF] from='client.? 172.21.15.132:0/615968305' entity='client.admin' 2024-04-25T19:53:56.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:53:56 smithi132 bash[18607]: audit 2024-04-25T19:53:55.618990+0000 mon.a (mon.0) 96 : audit [INF] from='client.? 172.21.15.132:0/615968305' entity='client.admin' 2024-04-25T19:54:01.148 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:54:04.955 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-25T19:54:04.955 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-25T19:54:05.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:04 smithi132 bash[18607]: audit 2024-04-25T19:54:03.858341+0000 mon.a (mon.0) 97 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:05.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:04 smithi132 bash[18607]: audit 2024-04-25T19:54:03.858341+0000 mon.a (mon.0) 97 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:05.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:04 smithi132 bash[18607]: audit 2024-04-25T19:54:03.864301+0000 mon.a (mon.0) 98 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:05.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:04 smithi132 bash[18607]: audit 2024-04-25T19:54:03.864301+0000 mon.a (mon.0) 98 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:05.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:04 smithi132 bash[18607]: audit 2024-04-25T19:54:03.866030+0000 mon.a (mon.0) 99 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi132", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:54:05.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:04 smithi132 bash[18607]: audit 2024-04-25T19:54:03.866030+0000 mon.a (mon.0) 99 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi132", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:54:05.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:04 smithi132 bash[18607]: audit 2024-04-25T19:54:03.873230+0000 mon.a (mon.0) 100 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:05.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:04 smithi132 bash[18607]: audit 2024-04-25T19:54:03.873230+0000 mon.a (mon.0) 100 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:05.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:04 smithi132 bash[18607]: audit 2024-04-25T19:54:03.907379+0000 mon.a (mon.0) 101 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:54:05.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:04 smithi132 bash[18607]: audit 2024-04-25T19:54:03.907379+0000 mon.a (mon.0) 101 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:54:05.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:04 smithi132 bash[18607]: audit 2024-04-25T19:54:03.912164+0000 mon.a (mon.0) 102 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:05.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:04 smithi132 bash[18607]: audit 2024-04-25T19:54:03.912164+0000 mon.a (mon.0) 102 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:05.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:04 smithi132 bash[18607]: audit 2024-04-25T19:54:04.307357+0000 mon.a (mon.0) 103 : audit [INF] from='client.? 172.21.15.132:0/4026289968' entity='client.admin' 2024-04-25T19:54:05.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:04 smithi132 bash[18607]: audit 2024-04-25T19:54:04.307357+0000 mon.a (mon.0) 103 : audit [INF] from='client.? 172.21.15.132:0/4026289968' entity='client.admin' 2024-04-25T19:54:05.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:04 smithi132 bash[18607]: audit 2024-04-25T19:54:04.359824+0000 mon.a (mon.0) 104 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:54:05.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:04 smithi132 bash[18607]: audit 2024-04-25T19:54:04.359824+0000 mon.a (mon.0) 104 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:54:05.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:04 smithi132 bash[18607]: audit 2024-04-25T19:54:04.364278+0000 mon.a (mon.0) 105 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:05.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:04 smithi132 bash[18607]: audit 2024-04-25T19:54:04.364278+0000 mon.a (mon.0) 105 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:06.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:05 smithi132 bash[18607]: cluster 2024-04-25T19:54:04.692714+0000 mgr.x (mgr.14152) 9 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:06.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:05 smithi132 bash[18607]: cluster 2024-04-25T19:54:04.692714+0000 mgr.x (mgr.14152) 9 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:08.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:07 smithi132 bash[18607]: cluster 2024-04-25T19:54:06.693097+0000 mgr.x (mgr.14152) 10 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:08.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:07 smithi132 bash[18607]: cluster 2024-04-25T19:54:06.693097+0000 mgr.x (mgr.14152) 10 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:09.672 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:54:10.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:09 smithi132 bash[18607]: cluster 2024-04-25T19:54:08.693514+0000 mgr.x (mgr.14152) 11 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:10.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:09 smithi132 bash[18607]: cluster 2024-04-25T19:54:08.693514+0000 mgr.x (mgr.14152) 11 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:12.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:11 smithi132 bash[18607]: cluster 2024-04-25T19:54:10.693910+0000 mgr.x (mgr.14152) 12 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:12.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:11 smithi132 bash[18607]: cluster 2024-04-25T19:54:10.693910+0000 mgr.x (mgr.14152) 12 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:13.019 INFO:tasks.cephadm:Remote smithi150 excluded from cephadm cluster by role 2024-04-25T19:54:13.019 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-25T19:54:13.019 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-25T19:54:13.774 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:13 smithi132 bash[18607]: audit 2024-04-25T19:54:12.387847+0000 mgr.x (mgr.14152) 13 : audit [DBG] from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:54:13.774 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:13 smithi132 bash[18607]: audit 2024-04-25T19:54:12.387847+0000 mgr.x (mgr.14152) 13 : audit [DBG] from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:54:13.774 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:13 smithi132 bash[18607]: audit 2024-04-25T19:54:12.392864+0000 mon.a (mon.0) 106 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:13.774 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:13 smithi132 bash[18607]: audit 2024-04-25T19:54:12.392864+0000 mon.a (mon.0) 106 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:13.774 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:13 smithi132 bash[18607]: audit 2024-04-25T19:54:12.419743+0000 mon.a (mon.0) 107 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:54:13.774 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:13 smithi132 bash[18607]: audit 2024-04-25T19:54:12.419743+0000 mon.a (mon.0) 107 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:54:13.774 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:13 smithi132 bash[18607]: audit 2024-04-25T19:54:12.421149+0000 mon.a (mon.0) 108 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:54:13.774 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:13 smithi132 bash[18607]: audit 2024-04-25T19:54:12.421149+0000 mon.a (mon.0) 108 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:54:13.774 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:13 smithi132 bash[18607]: audit 2024-04-25T19:54:12.421834+0000 mon.a (mon.0) 109 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:54:13.775 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:13 smithi132 bash[18607]: audit 2024-04-25T19:54:12.421834+0000 mon.a (mon.0) 109 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:54:13.775 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:13 smithi132 bash[18607]: cephadm 2024-04-25T19:54:12.422778+0000 mgr.x (mgr.14152) 14 : cephadm [INF] Updating smithi132:/etc/ceph/ceph.conf 2024-04-25T19:54:13.775 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:13 smithi132 bash[18607]: cephadm 2024-04-25T19:54:12.422778+0000 mgr.x (mgr.14152) 14 : cephadm [INF] Updating smithi132:/etc/ceph/ceph.conf 2024-04-25T19:54:13.775 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:13 smithi132 bash[18607]: cephadm 2024-04-25T19:54:12.536998+0000 mgr.x (mgr.14152) 15 : cephadm [INF] Updating smithi132:/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T19:54:13.775 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:13 smithi132 bash[18607]: cephadm 2024-04-25T19:54:12.536998+0000 mgr.x (mgr.14152) 15 : cephadm [INF] Updating smithi132:/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T19:54:13.775 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:13 smithi132 bash[18607]: cephadm 2024-04-25T19:54:12.651823+0000 mgr.x (mgr.14152) 16 : cephadm [INF] Updating smithi132:/etc/ceph/ceph.client.admin.keyring 2024-04-25T19:54:13.775 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:13 smithi132 bash[18607]: cephadm 2024-04-25T19:54:12.651823+0000 mgr.x (mgr.14152) 16 : cephadm [INF] Updating smithi132:/etc/ceph/ceph.client.admin.keyring 2024-04-25T19:54:13.775 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:13 smithi132 bash[18607]: cluster 2024-04-25T19:54:12.694303+0000 mgr.x (mgr.14152) 17 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:13.775 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:13 smithi132 bash[18607]: cluster 2024-04-25T19:54:12.694303+0000 mgr.x (mgr.14152) 17 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:13.775 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:13 smithi132 bash[18607]: cephadm 2024-04-25T19:54:12.766029+0000 mgr.x (mgr.14152) 18 : cephadm [INF] Updating smithi132:/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T19:54:13.775 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:13 smithi132 bash[18607]: cephadm 2024-04-25T19:54:12.766029+0000 mgr.x (mgr.14152) 18 : cephadm [INF] Updating smithi132:/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T19:54:13.775 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:13 smithi132 bash[18607]: audit 2024-04-25T19:54:12.883325+0000 mon.a (mon.0) 110 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:13.775 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:13 smithi132 bash[18607]: audit 2024-04-25T19:54:12.883325+0000 mon.a (mon.0) 110 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:13.775 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:13 smithi132 bash[18607]: audit 2024-04-25T19:54:12.886271+0000 mon.a (mon.0) 111 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:13.776 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:13 smithi132 bash[18607]: audit 2024-04-25T19:54:12.886271+0000 mon.a (mon.0) 111 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:13.776 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:13 smithi132 bash[18607]: audit 2024-04-25T19:54:12.889080+0000 mon.a (mon.0) 112 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:13.776 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:13 smithi132 bash[18607]: audit 2024-04-25T19:54:12.889080+0000 mon.a (mon.0) 112 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:16.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:15 smithi132 bash[18607]: cluster 2024-04-25T19:54:14.694751+0000 mgr.x (mgr.14152) 19 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:16.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:15 smithi132 bash[18607]: cluster 2024-04-25T19:54:14.694751+0000 mgr.x (mgr.14152) 19 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:17.726 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:54:18.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:17 smithi132 bash[18607]: cluster 2024-04-25T19:54:16.695185+0000 mgr.x (mgr.14152) 20 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:18.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:17 smithi132 bash[18607]: cluster 2024-04-25T19:54:16.695185+0000 mgr.x (mgr.14152) 20 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:20.004 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:19 smithi132 bash[18607]: cluster 2024-04-25T19:54:18.695550+0000 mgr.x (mgr.14152) 21 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:20.004 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:19 smithi132 bash[18607]: cluster 2024-04-25T19:54:18.695550+0000 mgr.x (mgr.14152) 21 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:20.762 INFO:teuthology.orchestra.run.smithi132.stderr:adjusted tunables profile to default 2024-04-25T19:54:21.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:20 smithi132 bash[18607]: audit 2024-04-25T19:54:20.387646+0000 mon.a (mon.0) 113 : audit [INF] from='client.? 172.21.15.132:0/1132336527' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-25T19:54:21.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:20 smithi132 bash[18607]: audit 2024-04-25T19:54:20.387646+0000 mon.a (mon.0) 113 : audit [INF] from='client.? 172.21.15.132:0/1132336527' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-25T19:54:21.393 INFO:tasks.cephadm:Adding mon.a on smithi132 2024-04-25T19:54:21.393 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch apply mon '1;smithi132:172.21.15.132=a' 2024-04-25T19:54:22.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:21 smithi132 bash[18607]: cluster 2024-04-25T19:54:20.696004+0000 mgr.x (mgr.14152) 22 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:22.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:21 smithi132 bash[18607]: cluster 2024-04-25T19:54:20.696004+0000 mgr.x (mgr.14152) 22 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:22.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:21 smithi132 bash[18607]: audit 2024-04-25T19:54:20.757296+0000 mon.a (mon.0) 114 : audit [INF] from='client.? 172.21.15.132:0/1132336527' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-25T19:54:22.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:21 smithi132 bash[18607]: audit 2024-04-25T19:54:20.757296+0000 mon.a (mon.0) 114 : audit [INF] from='client.? 172.21.15.132:0/1132336527' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-25T19:54:22.024 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:21 smithi132 bash[18607]: cluster 2024-04-25T19:54:20.761696+0000 mon.a (mon.0) 115 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-04-25T19:54:22.024 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:21 smithi132 bash[18607]: cluster 2024-04-25T19:54:20.761696+0000 mon.a (mon.0) 115 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-04-25T19:54:24.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:23 smithi132 bash[18607]: cluster 2024-04-25T19:54:22.696403+0000 mgr.x (mgr.14152) 23 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:24.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:23 smithi132 bash[18607]: cluster 2024-04-25T19:54:22.696403+0000 mgr.x (mgr.14152) 23 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:26.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:25 smithi132 bash[18607]: cluster 2024-04-25T19:54:24.696886+0000 mgr.x (mgr.14152) 24 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:26.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:25 smithi132 bash[18607]: cluster 2024-04-25T19:54:24.696886+0000 mgr.x (mgr.14152) 24 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:26.105 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:54:28.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:27 smithi132 bash[18607]: cluster 2024-04-25T19:54:26.697299+0000 mgr.x (mgr.14152) 25 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:28.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:27 smithi132 bash[18607]: cluster 2024-04-25T19:54:26.697299+0000 mgr.x (mgr.14152) 25 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:28.716 INFO:teuthology.orchestra.run.smithi132.stdout:Scheduled mon update... 2024-04-25T19:54:29.523 INFO:tasks.cephadm:Waiting for 1 mons in monmap... 2024-04-25T19:54:29.523 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T19:54:29.774 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:29 smithi132 bash[18607]: cluster 2024-04-25T19:54:28.697728+0000 mgr.x (mgr.14152) 26 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:29.774 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:29 smithi132 bash[18607]: cluster 2024-04-25T19:54:28.697728+0000 mgr.x (mgr.14152) 26 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:29.774 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:29 smithi132 bash[18607]: audit 2024-04-25T19:54:28.709258+0000 mgr.x (mgr.14152) 27 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "1;smithi132:172.21.15.132=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:54:29.774 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:29 smithi132 bash[18607]: audit 2024-04-25T19:54:28.709258+0000 mgr.x (mgr.14152) 27 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "1;smithi132:172.21.15.132=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:54:29.774 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:29 smithi132 bash[18607]: cephadm 2024-04-25T19:54:28.714196+0000 mgr.x (mgr.14152) 28 : cephadm [INF] Saving service mon spec with placement smithi132:172.21.15.132=a;count:1 2024-04-25T19:54:29.774 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:29 smithi132 bash[18607]: cephadm 2024-04-25T19:54:28.714196+0000 mgr.x (mgr.14152) 28 : cephadm [INF] Saving service mon spec with placement smithi132:172.21.15.132=a;count:1 2024-04-25T19:54:29.774 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:29 smithi132 bash[18607]: audit 2024-04-25T19:54:28.717256+0000 mon.a (mon.0) 116 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:29.774 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:29 smithi132 bash[18607]: audit 2024-04-25T19:54:28.717256+0000 mon.a (mon.0) 116 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:29.774 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:29 smithi132 bash[18607]: audit 2024-04-25T19:54:28.757749+0000 mon.a (mon.0) 117 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:54:29.774 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:29 smithi132 bash[18607]: audit 2024-04-25T19:54:28.757749+0000 mon.a (mon.0) 117 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:54:29.774 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:29 smithi132 bash[18607]: audit 2024-04-25T19:54:28.759150+0000 mon.a (mon.0) 118 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:54:29.775 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:29 smithi132 bash[18607]: audit 2024-04-25T19:54:28.759150+0000 mon.a (mon.0) 118 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:54:29.775 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:29 smithi132 bash[18607]: audit 2024-04-25T19:54:28.759694+0000 mon.a (mon.0) 119 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:54:29.775 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:29 smithi132 bash[18607]: audit 2024-04-25T19:54:28.759694+0000 mon.a (mon.0) 119 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:54:29.775 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:29 smithi132 bash[18607]: audit 2024-04-25T19:54:28.762932+0000 mon.a (mon.0) 120 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:29.775 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:29 smithi132 bash[18607]: audit 2024-04-25T19:54:28.762932+0000 mon.a (mon.0) 120 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:29.775 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:29 smithi132 bash[18607]: audit 2024-04-25T19:54:28.767186+0000 mon.a (mon.0) 121 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:29.775 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:29 smithi132 bash[18607]: audit 2024-04-25T19:54:28.767186+0000 mon.a (mon.0) 121 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:29.775 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:29 smithi132 bash[18607]: audit 2024-04-25T19:54:28.785435+0000 mon.a (mon.0) 122 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:29.775 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:29 smithi132 bash[18607]: audit 2024-04-25T19:54:28.785435+0000 mon.a (mon.0) 122 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:29.775 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:29 smithi132 bash[18607]: audit 2024-04-25T19:54:28.787232+0000 mon.a (mon.0) 123 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:29.775 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:29 smithi132 bash[18607]: audit 2024-04-25T19:54:28.787232+0000 mon.a (mon.0) 123 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:29.775 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:29 smithi132 bash[18607]: audit 2024-04-25T19:54:28.788970+0000 mon.a (mon.0) 124 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:29.775 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:29 smithi132 bash[18607]: audit 2024-04-25T19:54:28.788970+0000 mon.a (mon.0) 124 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:29.776 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:29 smithi132 bash[18607]: audit 2024-04-25T19:54:28.790606+0000 mon.a (mon.0) 125 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:29.776 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:29 smithi132 bash[18607]: audit 2024-04-25T19:54:28.790606+0000 mon.a (mon.0) 125 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:29.776 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:29 smithi132 bash[18607]: cephadm 2024-04-25T19:54:28.790882+0000 mgr.x (mgr.14152) 29 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2024-04-25T19:54:29.776 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:29 smithi132 bash[18607]: cephadm 2024-04-25T19:54:28.790882+0000 mgr.x (mgr.14152) 29 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2024-04-25T19:54:29.776 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:29 smithi132 bash[18607]: audit 2024-04-25T19:54:28.791114+0000 mon.a (mon.0) 126 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T19:54:29.776 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:29 smithi132 bash[18607]: audit 2024-04-25T19:54:28.791114+0000 mon.a (mon.0) 126 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T19:54:29.776 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:29 smithi132 bash[18607]: audit 2024-04-25T19:54:28.791643+0000 mon.a (mon.0) 127 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T19:54:29.776 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:29 smithi132 bash[18607]: audit 2024-04-25T19:54:28.791643+0000 mon.a (mon.0) 127 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T19:54:29.776 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:29 smithi132 bash[18607]: audit 2024-04-25T19:54:28.792149+0000 mon.a (mon.0) 128 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:54:29.776 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:29 smithi132 bash[18607]: audit 2024-04-25T19:54:28.792149+0000 mon.a (mon.0) 128 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:54:29.776 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:29 smithi132 bash[18607]: cephadm 2024-04-25T19:54:28.792875+0000 mgr.x (mgr.14152) 30 : cephadm [INF] Reconfiguring daemon mon.a on smithi132 2024-04-25T19:54:29.776 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:29 smithi132 bash[18607]: cephadm 2024-04-25T19:54:28.792875+0000 mgr.x (mgr.14152) 30 : cephadm [INF] Reconfiguring daemon mon.a on smithi132 2024-04-25T19:54:32.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:31 smithi132 bash[18607]: cluster 2024-04-25T19:54:30.698202+0000 mgr.x (mgr.14152) 31 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:32.024 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:31 smithi132 bash[18607]: cluster 2024-04-25T19:54:30.698202+0000 mgr.x (mgr.14152) 31 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:32.024 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:31 smithi132 bash[18607]: audit 2024-04-25T19:54:30.759874+0000 mon.a (mon.0) 129 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:32.024 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:31 smithi132 bash[18607]: audit 2024-04-25T19:54:30.759874+0000 mon.a (mon.0) 129 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:32.024 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:31 smithi132 bash[18607]: audit 2024-04-25T19:54:30.764995+0000 mon.a (mon.0) 130 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:32.024 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:31 smithi132 bash[18607]: audit 2024-04-25T19:54:30.764995+0000 mon.a (mon.0) 130 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:34.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:33 smithi132 bash[18607]: cluster 2024-04-25T19:54:32.698671+0000 mgr.x (mgr.14152) 32 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:34.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:33 smithi132 bash[18607]: cluster 2024-04-25T19:54:32.698671+0000 mgr.x (mgr.14152) 32 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:34.222 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:54:36.026 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:35 smithi132 bash[18607]: cluster 2024-04-25T19:54:34.699159+0000 mgr.x (mgr.14152) 33 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:36.026 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:35 smithi132 bash[18607]: cluster 2024-04-25T19:54:34.699159+0000 mgr.x (mgr.14152) 33 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:36.883 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T19:54:36.883 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"3114c848-033d-11ef-bc93-c7b262605968","modified":"2024-04-25T19:52:33.352311Z","created":"2024-04-25T19:52:33.352311Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-25T19:54:36.886 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-04-25T19:54:37.532 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-25T19:54:37.532 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-25T19:54:38.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:37 smithi132 bash[18607]: cluster 2024-04-25T19:54:36.699625+0000 mgr.x (mgr.14152) 34 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:38.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:37 smithi132 bash[18607]: cluster 2024-04-25T19:54:36.699625+0000 mgr.x (mgr.14152) 34 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:38.024 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:37 smithi132 bash[18607]: audit 2024-04-25T19:54:36.884372+0000 mon.a (mon.0) 131 : audit [DBG] from='client.? 172.21.15.132:0/200938204' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T19:54:38.024 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:37 smithi132 bash[18607]: audit 2024-04-25T19:54:36.884372+0000 mon.a (mon.0) 131 : audit [DBG] from='client.? 172.21.15.132:0/200938204' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T19:54:40.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:39 smithi132 bash[18607]: cluster 2024-04-25T19:54:38.700092+0000 mgr.x (mgr.14152) 35 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:40.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:39 smithi132 bash[18607]: cluster 2024-04-25T19:54:38.700092+0000 mgr.x (mgr.14152) 35 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:42.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:41 smithi132 bash[18607]: cluster 2024-04-25T19:54:40.700484+0000 mgr.x (mgr.14152) 36 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:42.024 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:41 smithi132 bash[18607]: cluster 2024-04-25T19:54:40.700484+0000 mgr.x (mgr.14152) 36 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:42.231 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:54:44.024 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:43 smithi132 bash[18607]: cluster 2024-04-25T19:54:42.700965+0000 mgr.x (mgr.14152) 37 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:44.024 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:43 smithi132 bash[18607]: cluster 2024-04-25T19:54:42.700965+0000 mgr.x (mgr.14152) 37 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:44.791 INFO:teuthology.orchestra.run.smithi132.stdout:# minimal ceph.conf for 3114c848-033d-11ef-bc93-c7b262605968 2024-04-25T19:54:44.792 INFO:teuthology.orchestra.run.smithi132.stdout:[global] 2024-04-25T19:54:44.792 INFO:teuthology.orchestra.run.smithi132.stdout: fsid = 3114c848-033d-11ef-bc93-c7b262605968 2024-04-25T19:54:44.792 INFO:teuthology.orchestra.run.smithi132.stdout: mon_host = [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] 2024-04-25T19:54:45.419 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-25T19:54:45.420 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-25T19:54:45.420 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/ceph/ceph.conf 2024-04-25T19:54:45.433 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-25T19:54:45.433 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T19:54:45.487 INFO:tasks.cephadm:Adding mgr.x on smithi132 2024-04-25T19:54:45.488 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch apply mgr '1;smithi132=x' 2024-04-25T19:54:46.024 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:45 smithi132 bash[18607]: cluster 2024-04-25T19:54:44.701411+0000 mgr.x (mgr.14152) 38 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:46.024 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:45 smithi132 bash[18607]: cluster 2024-04-25T19:54:44.701411+0000 mgr.x (mgr.14152) 38 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:46.024 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:45 smithi132 bash[18607]: audit 2024-04-25T19:54:44.792066+0000 mon.a (mon.0) 132 : audit [DBG] from='client.? 172.21.15.132:0/2998074396' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:54:46.024 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:45 smithi132 bash[18607]: audit 2024-04-25T19:54:44.792066+0000 mon.a (mon.0) 132 : audit [DBG] from='client.? 172.21.15.132:0/2998074396' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:54:48.024 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:47 smithi132 bash[18607]: cluster 2024-04-25T19:54:46.701868+0000 mgr.x (mgr.14152) 39 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:48.024 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:47 smithi132 bash[18607]: cluster 2024-04-25T19:54:46.701868+0000 mgr.x (mgr.14152) 39 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:50.024 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:49 smithi132 bash[18607]: cluster 2024-04-25T19:54:48.702310+0000 mgr.x (mgr.14152) 40 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:50.024 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:49 smithi132 bash[18607]: cluster 2024-04-25T19:54:48.702310+0000 mgr.x (mgr.14152) 40 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:50.239 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:54:52.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:51 smithi132 bash[18607]: cluster 2024-04-25T19:54:50.702682+0000 mgr.x (mgr.14152) 41 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:52.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:51 smithi132 bash[18607]: cluster 2024-04-25T19:54:50.702682+0000 mgr.x (mgr.14152) 41 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:52.939 INFO:teuthology.orchestra.run.smithi132.stdout:Scheduled mgr update... 2024-04-25T19:54:53.544 INFO:tasks.cephadm:Deploying OSDs... 2024-04-25T19:54:53.545 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-25T19:54:53.545 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T19:54:53.551 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T19:54:53.551 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_1 2024-04-25T19:54:53.600 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T19:54:53.600 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T19:54:53.601 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 5h/5d Inode: 884 Links: 1 2024-04-25T19:54:53.601 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T19:54:53.601 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2024-04-25 19:46:37.633501637 +0000 2024-04-25T19:54:53.601 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-04-25 19:46:37.349504174 +0000 2024-04-25T19:54:53.601 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-04-25 19:46:37.349504174 +0000 2024-04-25T19:54:53.601 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: - 2024-04-25T19:54:53.601 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T19:54:53.655 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2024-04-25T19:54:53.655 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2024-04-25T19:54:53.655 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.000416841 s, 1.2 MB/s 2024-04-25T19:54:53.656 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T19:54:53.705 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_2 2024-04-25T19:54:53.752 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T19:54:53.752 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T19:54:53.752 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 5h/5d Inode: 903 Links: 1 2024-04-25T19:54:53.752 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T19:54:53.752 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2024-04-25 19:46:38.113497350 +0000 2024-04-25T19:54:53.752 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-04-25 19:46:37.845499744 +0000 2024-04-25T19:54:53.752 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-04-25 19:46:37.845499744 +0000 2024-04-25T19:54:53.752 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: - 2024-04-25T19:54:53.752 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T19:54:53.806 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2024-04-25T19:54:53.806 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2024-04-25T19:54:53.806 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.000439029 s, 1.2 MB/s 2024-04-25T19:54:53.807 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T19:54:53.857 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_3 2024-04-25T19:54:53.904 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T19:54:53.905 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T19:54:53.905 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 5h/5d Inode: 919 Links: 1 2024-04-25T19:54:53.905 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T19:54:53.905 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2024-04-25 19:46:38.621492811 +0000 2024-04-25T19:54:53.905 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-04-25 19:46:38.333495384 +0000 2024-04-25T19:54:53.905 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-04-25 19:46:38.333495384 +0000 2024-04-25T19:54:53.905 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: - 2024-04-25T19:54:53.906 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T19:54:53.957 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2024-04-25T19:54:53.957 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2024-04-25T19:54:53.957 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.000280554 s, 1.8 MB/s 2024-04-25T19:54:53.958 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T19:54:54.005 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_4 2024-04-25T19:54:54.052 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T19:54:54.054 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T19:54:54.054 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 5h/5d Inode: 934 Links: 1 2024-04-25T19:54:54.054 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T19:54:54.054 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2024-04-25 19:46:39.125488310 +0000 2024-04-25T19:54:54.054 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-04-25 19:46:38.857490703 +0000 2024-04-25T19:54:54.055 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-04-25 19:46:38.857490703 +0000 2024-04-25T19:54:54.055 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: - 2024-04-25T19:54:54.056 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T19:54:54.103 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:53 smithi132 bash[18607]: cluster 2024-04-25T19:54:52.702944+0000 mgr.x (mgr.14152) 42 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:54.103 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:53 smithi132 bash[18607]: cluster 2024-04-25T19:54:52.702944+0000 mgr.x (mgr.14152) 42 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:54.103 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:53 smithi132 bash[18607]: audit 2024-04-25T19:54:52.930366+0000 mgr.x (mgr.14152) 43 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "1;smithi132=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:54:54.104 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:53 smithi132 bash[18607]: audit 2024-04-25T19:54:52.930366+0000 mgr.x (mgr.14152) 43 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "1;smithi132=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:54:54.104 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:53 smithi132 bash[18607]: cephadm 2024-04-25T19:54:52.932214+0000 mgr.x (mgr.14152) 44 : cephadm [INF] Saving service mgr spec with placement smithi132=x;count:1 2024-04-25T19:54:54.104 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:53 smithi132 bash[18607]: cephadm 2024-04-25T19:54:52.932214+0000 mgr.x (mgr.14152) 44 : cephadm [INF] Saving service mgr spec with placement smithi132=x;count:1 2024-04-25T19:54:54.104 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:53 smithi132 bash[18607]: audit 2024-04-25T19:54:52.939141+0000 mon.a (mon.0) 133 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:54.104 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:53 smithi132 bash[18607]: audit 2024-04-25T19:54:52.939141+0000 mon.a (mon.0) 133 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:54.104 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:53 smithi132 bash[18607]: audit 2024-04-25T19:54:52.983071+0000 mon.a (mon.0) 134 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:54:54.104 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:53 smithi132 bash[18607]: audit 2024-04-25T19:54:52.983071+0000 mon.a (mon.0) 134 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:54:54.104 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:53 smithi132 bash[18607]: audit 2024-04-25T19:54:53.561156+0000 mon.a (mon.0) 135 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:54:54.104 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:53 smithi132 bash[18607]: audit 2024-04-25T19:54:53.561156+0000 mon.a (mon.0) 135 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:54:54.104 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:53 smithi132 bash[18607]: audit 2024-04-25T19:54:53.562264+0000 mon.a (mon.0) 136 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:54:54.104 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:53 smithi132 bash[18607]: audit 2024-04-25T19:54:53.562264+0000 mon.a (mon.0) 136 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:54:54.104 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:53 smithi132 bash[18607]: audit 2024-04-25T19:54:53.566713+0000 mon.a (mon.0) 137 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:54.104 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:53 smithi132 bash[18607]: audit 2024-04-25T19:54:53.566713+0000 mon.a (mon.0) 137 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:54.105 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:53 smithi132 bash[18607]: audit 2024-04-25T19:54:53.573396+0000 mon.a (mon.0) 138 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:54.105 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:53 smithi132 bash[18607]: audit 2024-04-25T19:54:53.573396+0000 mon.a (mon.0) 138 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:54.105 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:53 smithi132 bash[18607]: audit 2024-04-25T19:54:53.607785+0000 mon.a (mon.0) 139 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T19:54:54.105 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:53 smithi132 bash[18607]: audit 2024-04-25T19:54:53.607785+0000 mon.a (mon.0) 139 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T19:54:54.105 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:53 smithi132 bash[18607]: audit 2024-04-25T19:54:53.608973+0000 mon.a (mon.0) 140 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T19:54:54.105 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:53 smithi132 bash[18607]: audit 2024-04-25T19:54:53.608973+0000 mon.a (mon.0) 140 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T19:54:54.105 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:53 smithi132 bash[18607]: audit 2024-04-25T19:54:53.610214+0000 mon.a (mon.0) 141 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:54:54.105 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:53 smithi132 bash[18607]: audit 2024-04-25T19:54:53.610214+0000 mon.a (mon.0) 141 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:54:54.106 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2024-04-25T19:54:54.106 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2024-04-25T19:54:54.106 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.000252416 s, 2.0 MB/s 2024-04-25T19:54:54.107 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T19:54:54.153 INFO:tasks.cephadm:Deploying osd.0 on smithi132 with /dev/vg_nvme/lv_4... 2024-04-25T19:54:54.153 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-25T19:54:55.244 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:54 smithi132 bash[18607]: cephadm 2024-04-25T19:54:53.607166+0000 mgr.x (mgr.14152) 45 : cephadm [INF] Reconfiguring mgr.x (unknown last config time)... 2024-04-25T19:54:55.244 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:54 smithi132 bash[18607]: cephadm 2024-04-25T19:54:53.607166+0000 mgr.x (mgr.14152) 45 : cephadm [INF] Reconfiguring mgr.x (unknown last config time)... 2024-04-25T19:54:55.244 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:54 smithi132 bash[18607]: cephadm 2024-04-25T19:54:53.611437+0000 mgr.x (mgr.14152) 46 : cephadm [INF] Reconfiguring daemon mgr.x on smithi132 2024-04-25T19:54:55.244 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:54 smithi132 bash[18607]: cephadm 2024-04-25T19:54:53.611437+0000 mgr.x (mgr.14152) 46 : cephadm [INF] Reconfiguring daemon mgr.x on smithi132 2024-04-25T19:54:56.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:55 smithi132 bash[18607]: cluster 2024-04-25T19:54:54.703291+0000 mgr.x (mgr.14152) 47 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:56.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:55 smithi132 bash[18607]: cluster 2024-04-25T19:54:54.703291+0000 mgr.x (mgr.14152) 47 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:56.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:55 smithi132 bash[18607]: audit 2024-04-25T19:54:55.676363+0000 mon.a (mon.0) 142 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:56.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:55 smithi132 bash[18607]: audit 2024-04-25T19:54:55.676363+0000 mon.a (mon.0) 142 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:56.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:55 smithi132 bash[18607]: audit 2024-04-25T19:54:55.684391+0000 mon.a (mon.0) 143 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:56.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:55 smithi132 bash[18607]: audit 2024-04-25T19:54:55.684391+0000 mon.a (mon.0) 143 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:54:57.896 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:54:58.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:57 smithi132 bash[18607]: cluster 2024-04-25T19:54:56.703792+0000 mgr.x (mgr.14152) 48 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:54:58.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:57 smithi132 bash[18607]: cluster 2024-04-25T19:54:56.703792+0000 mgr.x (mgr.14152) 48 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:00.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:59 smithi132 bash[18607]: cluster 2024-04-25T19:54:58.704163+0000 mgr.x (mgr.14152) 49 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:00.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:54:59 smithi132 bash[18607]: cluster 2024-04-25T19:54:58.704163+0000 mgr.x (mgr.14152) 49 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:00.743 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-25T19:55:00.743 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-25T19:55:00.743 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 10+0 records in 2024-04-25T19:55:00.743 INFO:teuthology.orchestra.run.smithi132.stderr:10+0 records out 2024-04-25T19:55:00.743 INFO:teuthology.orchestra.run.smithi132.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.01663 s, 631 MB/s 2024-04-25T19:55:00.743 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping successful for: 2024-04-25T19:55:01.365 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi132:vg_nvme/lv_4 2024-04-25T19:55:02.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:01 smithi132 bash[18607]: cluster 2024-04-25T19:55:00.704513+0000 mgr.x (mgr.14152) 50 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:02.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:01 smithi132 bash[18607]: cluster 2024-04-25T19:55:00.704513+0000 mgr.x (mgr.14152) 50 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:04.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:03 smithi132 bash[18607]: cluster 2024-04-25T19:55:02.704913+0000 mgr.x (mgr.14152) 51 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:04.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:03 smithi132 bash[18607]: cluster 2024-04-25T19:55:02.704913+0000 mgr.x (mgr.14152) 51 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:06.078 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:55:06.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:05 smithi132 bash[18607]: cluster 2024-04-25T19:55:04.705276+0000 mgr.x (mgr.14152) 52 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:06.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:05 smithi132 bash[18607]: cluster 2024-04-25T19:55:04.705276+0000 mgr.x (mgr.14152) 52 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:08.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:07 smithi132 bash[18607]: cluster 2024-04-25T19:55:06.705584+0000 mgr.x (mgr.14152) 53 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:08.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:07 smithi132 bash[18607]: cluster 2024-04-25T19:55:06.705584+0000 mgr.x (mgr.14152) 53 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:09.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:08 smithi132 bash[18607]: audit 2024-04-25T19:55:08.680808+0000 mon.a (mon.0) 144 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T19:55:09.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:08 smithi132 bash[18607]: audit 2024-04-25T19:55:08.680808+0000 mon.a (mon.0) 144 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T19:55:09.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:08 smithi132 bash[18607]: audit 2024-04-25T19:55:08.686990+0000 mon.a (mon.0) 145 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T19:55:09.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:08 smithi132 bash[18607]: audit 2024-04-25T19:55:08.686990+0000 mon.a (mon.0) 145 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T19:55:09.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:08 smithi132 bash[18607]: audit 2024-04-25T19:55:08.688060+0000 mon.a (mon.0) 146 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:55:09.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:08 smithi132 bash[18607]: audit 2024-04-25T19:55:08.688060+0000 mon.a (mon.0) 146 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:55:10.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:09 smithi132 bash[18607]: audit 2024-04-25T19:55:08.673456+0000 mgr.x (mgr.14152) 54 : audit [DBG] from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:55:10.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:09 smithi132 bash[18607]: audit 2024-04-25T19:55:08.673456+0000 mgr.x (mgr.14152) 54 : audit [DBG] from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:55:10.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:09 smithi132 bash[18607]: cluster 2024-04-25T19:55:08.705822+0000 mgr.x (mgr.14152) 55 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:10.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:09 smithi132 bash[18607]: cluster 2024-04-25T19:55:08.705822+0000 mgr.x (mgr.14152) 55 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:12.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:11 smithi132 bash[18607]: cluster 2024-04-25T19:55:10.706262+0000 mgr.x (mgr.14152) 56 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:12.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:11 smithi132 bash[18607]: cluster 2024-04-25T19:55:10.706262+0000 mgr.x (mgr.14152) 56 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:14.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:13 smithi132 bash[18607]: cluster 2024-04-25T19:55:12.706760+0000 mgr.x (mgr.14152) 57 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:14.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:13 smithi132 bash[18607]: cluster 2024-04-25T19:55:12.706760+0000 mgr.x (mgr.14152) 57 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:16.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:15 smithi132 bash[18607]: cluster 2024-04-25T19:55:14.707265+0000 mgr.x (mgr.14152) 58 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:16.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:15 smithi132 bash[18607]: cluster 2024-04-25T19:55:14.707265+0000 mgr.x (mgr.14152) 58 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:17.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:16 smithi132 bash[18607]: audit 2024-04-25T19:55:16.770485+0000 mon.a (mon.0) 147 : audit [INF] from='client.? 172.21.15.132:0/2371944404' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8345c5ff-03d5-4723-91dd-79e2d3685f55"}]: dispatch 2024-04-25T19:55:17.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:16 smithi132 bash[18607]: audit 2024-04-25T19:55:16.770485+0000 mon.a (mon.0) 147 : audit [INF] from='client.? 172.21.15.132:0/2371944404' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8345c5ff-03d5-4723-91dd-79e2d3685f55"}]: dispatch 2024-04-25T19:55:17.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:16 smithi132 bash[18607]: audit 2024-04-25T19:55:16.772278+0000 mon.a (mon.0) 148 : audit [INF] from='client.? 172.21.15.132:0/2371944404' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8345c5ff-03d5-4723-91dd-79e2d3685f55"}]': finished 2024-04-25T19:55:17.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:16 smithi132 bash[18607]: audit 2024-04-25T19:55:16.772278+0000 mon.a (mon.0) 148 : audit [INF] from='client.? 172.21.15.132:0/2371944404' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8345c5ff-03d5-4723-91dd-79e2d3685f55"}]': finished 2024-04-25T19:55:17.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:16 smithi132 bash[18607]: cluster 2024-04-25T19:55:16.777398+0000 mon.a (mon.0) 149 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2024-04-25T19:55:17.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:16 smithi132 bash[18607]: cluster 2024-04-25T19:55:16.777398+0000 mon.a (mon.0) 149 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2024-04-25T19:55:17.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:16 smithi132 bash[18607]: audit 2024-04-25T19:55:16.777697+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T19:55:17.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:16 smithi132 bash[18607]: audit 2024-04-25T19:55:16.777697+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T19:55:18.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:17 smithi132 bash[18607]: cluster 2024-04-25T19:55:16.707694+0000 mgr.x (mgr.14152) 59 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:18.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:17 smithi132 bash[18607]: cluster 2024-04-25T19:55:16.707694+0000 mgr.x (mgr.14152) 59 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:18.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:17 smithi132 bash[18607]: audit 2024-04-25T19:55:17.483389+0000 mon.a (mon.0) 151 : audit [DBG] from='client.? 172.21.15.132:0/2332042075' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T19:55:18.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:17 smithi132 bash[18607]: audit 2024-04-25T19:55:17.483389+0000 mon.a (mon.0) 151 : audit [DBG] from='client.? 172.21.15.132:0/2332042075' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T19:55:20.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:19 smithi132 bash[18607]: cluster 2024-04-25T19:55:18.708100+0000 mgr.x (mgr.14152) 60 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:20.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:19 smithi132 bash[18607]: cluster 2024-04-25T19:55:18.708100+0000 mgr.x (mgr.14152) 60 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:22.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:21 smithi132 bash[18607]: cluster 2024-04-25T19:55:20.708465+0000 mgr.x (mgr.14152) 61 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:22.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:21 smithi132 bash[18607]: cluster 2024-04-25T19:55:20.708465+0000 mgr.x (mgr.14152) 61 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:24.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:23 smithi132 bash[18607]: cluster 2024-04-25T19:55:22.708956+0000 mgr.x (mgr.14152) 62 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:24.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:23 smithi132 bash[18607]: cluster 2024-04-25T19:55:22.708956+0000 mgr.x (mgr.14152) 62 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:26.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:25 smithi132 bash[18607]: cluster 2024-04-25T19:55:24.709403+0000 mgr.x (mgr.14152) 63 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:26.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:25 smithi132 bash[18607]: cluster 2024-04-25T19:55:24.709403+0000 mgr.x (mgr.14152) 63 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:28.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:27 smithi132 bash[18607]: cluster 2024-04-25T19:55:26.709818+0000 mgr.x (mgr.14152) 64 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:28.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:27 smithi132 bash[18607]: cluster 2024-04-25T19:55:26.709818+0000 mgr.x (mgr.14152) 64 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:30.246 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:29 smithi132 bash[18607]: cluster 2024-04-25T19:55:28.710245+0000 mgr.x (mgr.14152) 65 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:30.246 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:29 smithi132 bash[18607]: cluster 2024-04-25T19:55:28.710245+0000 mgr.x (mgr.14152) 65 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:30.246 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:29 smithi132 bash[18607]: audit 2024-04-25T19:55:29.144485+0000 mon.a (mon.0) 152 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-25T19:55:30.246 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:29 smithi132 bash[18607]: audit 2024-04-25T19:55:29.144485+0000 mon.a (mon.0) 152 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-25T19:55:30.246 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:29 smithi132 bash[18607]: audit 2024-04-25T19:55:29.146298+0000 mon.a (mon.0) 153 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:55:30.246 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:29 smithi132 bash[18607]: audit 2024-04-25T19:55:29.146298+0000 mon.a (mon.0) 153 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:55:30.246 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:29 smithi132 bash[18607]: cephadm 2024-04-25T19:55:29.147747+0000 mgr.x (mgr.14152) 66 : cephadm [INF] Deploying daemon osd.0 on smithi132 2024-04-25T19:55:30.246 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:29 smithi132 bash[18607]: cephadm 2024-04-25T19:55:29.147747+0000 mgr.x (mgr.14152) 66 : cephadm [INF] Deploying daemon osd.0 on smithi132 2024-04-25T19:55:32.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:31 smithi132 bash[18607]: cluster 2024-04-25T19:55:30.710784+0000 mgr.x (mgr.14152) 67 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:32.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:31 smithi132 bash[18607]: cluster 2024-04-25T19:55:30.710784+0000 mgr.x (mgr.14152) 67 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:33.114 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:55:33 smithi132 systemd[1]: /etc/systemd/system/ceph-3114c848-033d-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-25T19:55:33.114 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:33 smithi132 systemd[1]: /etc/systemd/system/ceph-3114c848-033d-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-25T19:55:33.380 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:33 smithi132 systemd[1]: /etc/systemd/system/ceph-3114c848-033d-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-25T19:55:33.380 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:55:33 smithi132 systemd[1]: /etc/systemd/system/ceph-3114c848-033d-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-25T19:55:34.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:33 smithi132 bash[18607]: cluster 2024-04-25T19:55:32.711298+0000 mgr.x (mgr.14152) 68 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:34.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:33 smithi132 bash[18607]: cluster 2024-04-25T19:55:32.711298+0000 mgr.x (mgr.14152) 68 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:34.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:33 smithi132 bash[18607]: audit 2024-04-25T19:55:33.374660+0000 mon.a (mon.0) 154 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:55:34.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:33 smithi132 bash[18607]: audit 2024-04-25T19:55:33.374660+0000 mon.a (mon.0) 154 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:55:34.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:33 smithi132 bash[18607]: audit 2024-04-25T19:55:33.377880+0000 mon.a (mon.0) 155 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:55:34.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:33 smithi132 bash[18607]: audit 2024-04-25T19:55:33.377880+0000 mon.a (mon.0) 155 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:55:34.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:33 smithi132 bash[18607]: audit 2024-04-25T19:55:33.393571+0000 mon.a (mon.0) 156 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:55:34.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:33 smithi132 bash[18607]: audit 2024-04-25T19:55:33.393571+0000 mon.a (mon.0) 156 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:55:36.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:35 smithi132 bash[18607]: cluster 2024-04-25T19:55:34.711640+0000 mgr.x (mgr.14152) 69 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:36.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:35 smithi132 bash[18607]: cluster 2024-04-25T19:55:34.711640+0000 mgr.x (mgr.14152) 69 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:38.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:37 smithi132 bash[18607]: cluster 2024-04-25T19:55:36.712099+0000 mgr.x (mgr.14152) 70 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:38.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:37 smithi132 bash[18607]: cluster 2024-04-25T19:55:36.712099+0000 mgr.x (mgr.14152) 70 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:40.024 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:39 smithi132 bash[18607]: audit 2024-04-25T19:55:38.625394+0000 mon.a (mon.0) 157 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:55:40.025 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:39 smithi132 bash[18607]: audit 2024-04-25T19:55:38.625394+0000 mon.a (mon.0) 157 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:55:40.025 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:39 smithi132 bash[18607]: audit 2024-04-25T19:55:38.629247+0000 mon.a (mon.0) 158 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:55:40.025 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:39 smithi132 bash[18607]: audit 2024-04-25T19:55:38.629247+0000 mon.a (mon.0) 158 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:55:40.025 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:39 smithi132 bash[18607]: audit 2024-04-25T19:55:38.630181+0000 mon.a (mon.0) 159 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:55:40.025 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:39 smithi132 bash[18607]: audit 2024-04-25T19:55:38.630181+0000 mon.a (mon.0) 159 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:55:40.025 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:39 smithi132 bash[18607]: audit 2024-04-25T19:55:38.631240+0000 mon.a (mon.0) 160 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:55:40.025 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:39 smithi132 bash[18607]: audit 2024-04-25T19:55:38.631240+0000 mon.a (mon.0) 160 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:55:40.025 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:39 smithi132 bash[18607]: audit 2024-04-25T19:55:38.635440+0000 mon.a (mon.0) 161 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:55:40.025 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:39 smithi132 bash[18607]: audit 2024-04-25T19:55:38.635440+0000 mon.a (mon.0) 161 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:55:40.025 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:39 smithi132 bash[18607]: cluster 2024-04-25T19:55:38.712407+0000 mgr.x (mgr.14152) 71 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:40.025 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:39 smithi132 bash[18607]: cluster 2024-04-25T19:55:38.712407+0000 mgr.x (mgr.14152) 71 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:40.774 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:40 smithi132 bash[18607]: audit 2024-04-25T19:55:39.630493+0000 mon.a (mon.0) 162 : audit [INF] from='osd.0 [v2:172.21.15.132:6802/3788012097,v1:172.21.15.132:6803/3788012097]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T19:55:40.774 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:40 smithi132 bash[18607]: audit 2024-04-25T19:55:39.630493+0000 mon.a (mon.0) 162 : audit [INF] from='osd.0 [v2:172.21.15.132:6802/3788012097,v1:172.21.15.132:6803/3788012097]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T19:55:41.930 INFO:teuthology.orchestra.run.smithi132.stdout:Created osd(s) 0 on host 'smithi132' 2024-04-25T19:55:41.949 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:41 smithi132 bash[18607]: audit 2024-04-25T19:55:40.627599+0000 mon.a (mon.0) 163 : audit [INF] from='osd.0 [v2:172.21.15.132:6802/3788012097,v1:172.21.15.132:6803/3788012097]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-25T19:55:41.949 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:41 smithi132 bash[18607]: audit 2024-04-25T19:55:40.627599+0000 mon.a (mon.0) 163 : audit [INF] from='osd.0 [v2:172.21.15.132:6802/3788012097,v1:172.21.15.132:6803/3788012097]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-25T19:55:41.949 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:41 smithi132 bash[18607]: cluster 2024-04-25T19:55:40.629212+0000 mon.a (mon.0) 164 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-04-25T19:55:41.949 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:41 smithi132 bash[18607]: cluster 2024-04-25T19:55:40.629212+0000 mon.a (mon.0) 164 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-04-25T19:55:41.950 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:41 smithi132 bash[18607]: audit 2024-04-25T19:55:40.629442+0000 mon.a (mon.0) 165 : audit [INF] from='osd.0 [v2:172.21.15.132:6802/3788012097,v1:172.21.15.132:6803/3788012097]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-04-25T19:55:41.950 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:41 smithi132 bash[18607]: audit 2024-04-25T19:55:40.629442+0000 mon.a (mon.0) 165 : audit [INF] from='osd.0 [v2:172.21.15.132:6802/3788012097,v1:172.21.15.132:6803/3788012097]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-04-25T19:55:41.950 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:41 smithi132 bash[18607]: audit 2024-04-25T19:55:40.629778+0000 mon.a (mon.0) 166 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T19:55:41.950 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:41 smithi132 bash[18607]: audit 2024-04-25T19:55:40.629778+0000 mon.a (mon.0) 166 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T19:55:41.950 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:41 smithi132 bash[18607]: cluster 2024-04-25T19:55:40.712815+0000 mgr.x (mgr.14152) 72 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:41.950 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:41 smithi132 bash[18607]: cluster 2024-04-25T19:55:40.712815+0000 mgr.x (mgr.14152) 72 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:42.598 DEBUG:teuthology.orchestra.run.smithi132:osd.0> sudo journalctl -f -n 0 -u ceph-3114c848-033d-11ef-bc93-c7b262605968@osd.0.service 2024-04-25T19:55:42.601 INFO:tasks.cephadm:Deploying osd.1 on smithi132 with /dev/vg_nvme/lv_3... 2024-04-25T19:55:42.601 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-25T19:55:43.025 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:42 smithi132 bash[18607]: cluster 2024-04-25T19:55:40.665689+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-25T19:55:43.025 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:42 smithi132 bash[18607]: cluster 2024-04-25T19:55:40.665689+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-25T19:55:43.025 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:42 smithi132 bash[18607]: cluster 2024-04-25T19:55:40.665759+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-25T19:55:43.025 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:42 smithi132 bash[18607]: cluster 2024-04-25T19:55:40.665759+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-25T19:55:43.025 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:42 smithi132 bash[18607]: audit 2024-04-25T19:55:41.628867+0000 mon.a (mon.0) 167 : audit [INF] from='osd.0 [v2:172.21.15.132:6802/3788012097,v1:172.21.15.132:6803/3788012097]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-04-25T19:55:43.025 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:42 smithi132 bash[18607]: audit 2024-04-25T19:55:41.628867+0000 mon.a (mon.0) 167 : audit [INF] from='osd.0 [v2:172.21.15.132:6802/3788012097,v1:172.21.15.132:6803/3788012097]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-04-25T19:55:43.025 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:42 smithi132 bash[18607]: cluster 2024-04-25T19:55:41.633282+0000 mon.a (mon.0) 168 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-04-25T19:55:43.025 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:42 smithi132 bash[18607]: cluster 2024-04-25T19:55:41.633282+0000 mon.a (mon.0) 168 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-04-25T19:55:43.026 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:42 smithi132 bash[18607]: audit 2024-04-25T19:55:41.641357+0000 mon.a (mon.0) 169 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T19:55:43.026 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:42 smithi132 bash[18607]: audit 2024-04-25T19:55:41.641357+0000 mon.a (mon.0) 169 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T19:55:43.026 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:42 smithi132 bash[18607]: audit 2024-04-25T19:55:41.840949+0000 mon.a (mon.0) 170 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:55:43.026 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:42 smithi132 bash[18607]: audit 2024-04-25T19:55:41.840949+0000 mon.a (mon.0) 170 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:55:43.026 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:42 smithi132 bash[18607]: audit 2024-04-25T19:55:41.860136+0000 mon.a (mon.0) 171 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:55:43.026 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:42 smithi132 bash[18607]: audit 2024-04-25T19:55:41.860136+0000 mon.a (mon.0) 171 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:55:43.026 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:42 smithi132 bash[18607]: audit 2024-04-25T19:55:41.909898+0000 mon.a (mon.0) 172 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:55:43.026 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:42 smithi132 bash[18607]: audit 2024-04-25T19:55:41.909898+0000 mon.a (mon.0) 172 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:55:43.026 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:42 smithi132 bash[18607]: audit 2024-04-25T19:55:42.086309+0000 mon.a (mon.0) 173 : audit [INF] from='osd.0 [v2:172.21.15.132:6802/3788012097,v1:172.21.15.132:6803/3788012097]' entity='osd.0' 2024-04-25T19:55:43.026 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:42 smithi132 bash[18607]: audit 2024-04-25T19:55:42.086309+0000 mon.a (mon.0) 173 : audit [INF] from='osd.0 [v2:172.21.15.132:6802/3788012097,v1:172.21.15.132:6803/3788012097]' entity='osd.0' 2024-04-25T19:55:44.025 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:43 smithi132 bash[18607]: audit 2024-04-25T19:55:42.639149+0000 mon.a (mon.0) 174 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T19:55:44.026 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:43 smithi132 bash[18607]: audit 2024-04-25T19:55:42.639149+0000 mon.a (mon.0) 174 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T19:55:44.026 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:43 smithi132 bash[18607]: cluster 2024-04-25T19:55:42.713246+0000 mgr.x (mgr.14152) 73 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:44.026 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:43 smithi132 bash[18607]: cluster 2024-04-25T19:55:42.713246+0000 mgr.x (mgr.14152) 73 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T19:55:44.026 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:43 smithi132 bash[18607]: cluster 2024-04-25T19:55:43.089960+0000 mon.a (mon.0) 175 : cluster [INF] osd.0 [v2:172.21.15.132:6802/3788012097,v1:172.21.15.132:6803/3788012097] boot 2024-04-25T19:55:44.026 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:43 smithi132 bash[18607]: cluster 2024-04-25T19:55:43.089960+0000 mon.a (mon.0) 175 : cluster [INF] osd.0 [v2:172.21.15.132:6802/3788012097,v1:172.21.15.132:6803/3788012097] boot 2024-04-25T19:55:44.026 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:43 smithi132 bash[18607]: cluster 2024-04-25T19:55:43.090120+0000 mon.a (mon.0) 176 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2024-04-25T19:55:44.026 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:43 smithi132 bash[18607]: cluster 2024-04-25T19:55:43.090120+0000 mon.a (mon.0) 176 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2024-04-25T19:55:44.026 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:43 smithi132 bash[18607]: audit 2024-04-25T19:55:43.090863+0000 mon.a (mon.0) 177 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T19:55:44.026 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:43 smithi132 bash[18607]: audit 2024-04-25T19:55:43.090863+0000 mon.a (mon.0) 177 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T19:55:46.025 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:45 smithi132 bash[18607]: cluster 2024-04-25T19:55:44.648520+0000 mon.a (mon.0) 178 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-04-25T19:55:46.026 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:45 smithi132 bash[18607]: cluster 2024-04-25T19:55:44.648520+0000 mon.a (mon.0) 178 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-04-25T19:55:46.026 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:45 smithi132 bash[18607]: cluster 2024-04-25T19:55:44.713771+0000 mgr.x (mgr.14152) 74 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:55:46.026 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:45 smithi132 bash[18607]: cluster 2024-04-25T19:55:44.713771+0000 mgr.x (mgr.14152) 74 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:55:46.885 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:55:48.024 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:47 smithi132 bash[18607]: cluster 2024-04-25T19:55:46.714255+0000 mgr.x (mgr.14152) 75 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:55:48.024 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:47 smithi132 bash[18607]: cluster 2024-04-25T19:55:46.714255+0000 mgr.x (mgr.14152) 75 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:55:50.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:49 smithi132 bash[18607]: cluster 2024-04-25T19:55:48.714721+0000 mgr.x (mgr.14152) 76 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:55:50.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:49 smithi132 bash[18607]: cluster 2024-04-25T19:55:48.714721+0000 mgr.x (mgr.14152) 76 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:55:50.679 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-25T19:55:50.680 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-25T19:55:50.680 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 10+0 records in 2024-04-25T19:55:50.680 INFO:teuthology.orchestra.run.smithi132.stderr:10+0 records out 2024-04-25T19:55:50.680 INFO:teuthology.orchestra.run.smithi132.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0134323 s, 781 MB/s 2024-04-25T19:55:50.680 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping successful for: 2024-04-25T19:55:51.443 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi132:vg_nvme/lv_3 2024-04-25T19:55:52.024 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:51 smithi132 bash[18607]: cluster 2024-04-25T19:55:50.715048+0000 mgr.x (mgr.14152) 77 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:55:52.024 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:51 smithi132 bash[18607]: cluster 2024-04-25T19:55:50.715048+0000 mgr.x (mgr.14152) 77 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:55:53.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:52 smithi132 bash[18607]: cephadm 2024-04-25T19:55:51.780551+0000 mgr.x (mgr.14152) 78 : cephadm [INF] Detected new or changed devices on smithi132 2024-04-25T19:55:53.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:52 smithi132 bash[18607]: cephadm 2024-04-25T19:55:51.780551+0000 mgr.x (mgr.14152) 78 : cephadm [INF] Detected new or changed devices on smithi132 2024-04-25T19:55:53.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:52 smithi132 bash[18607]: audit 2024-04-25T19:55:51.788588+0000 mon.a (mon.0) 179 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:55:53.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:52 smithi132 bash[18607]: audit 2024-04-25T19:55:51.788588+0000 mon.a (mon.0) 179 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:55:53.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:52 smithi132 bash[18607]: audit 2024-04-25T19:55:51.795135+0000 mon.a (mon.0) 180 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:55:53.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:52 smithi132 bash[18607]: audit 2024-04-25T19:55:51.795135+0000 mon.a (mon.0) 180 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:55:53.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:52 smithi132 bash[18607]: audit 2024-04-25T19:55:51.796913+0000 mon.a (mon.0) 181 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:55:53.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:52 smithi132 bash[18607]: audit 2024-04-25T19:55:51.796913+0000 mon.a (mon.0) 181 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:55:53.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:52 smithi132 bash[18607]: cephadm 2024-04-25T19:55:51.798089+0000 mgr.x (mgr.14152) 79 : cephadm [INF] Adjusting osd_memory_target on smithi132 to 17265M 2024-04-25T19:55:53.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:52 smithi132 bash[18607]: cephadm 2024-04-25T19:55:51.798089+0000 mgr.x (mgr.14152) 79 : cephadm [INF] Adjusting osd_memory_target on smithi132 to 17265M 2024-04-25T19:55:53.276 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:52 smithi132 bash[18607]: audit 2024-04-25T19:55:51.802390+0000 mon.a (mon.0) 182 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:55:53.276 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:52 smithi132 bash[18607]: audit 2024-04-25T19:55:51.802390+0000 mon.a (mon.0) 182 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:55:53.276 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:52 smithi132 bash[18607]: audit 2024-04-25T19:55:51.805994+0000 mon.a (mon.0) 183 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:55:53.276 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:52 smithi132 bash[18607]: audit 2024-04-25T19:55:51.805994+0000 mon.a (mon.0) 183 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:55:53.276 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:52 smithi132 bash[18607]: audit 2024-04-25T19:55:51.807477+0000 mon.a (mon.0) 184 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:55:53.276 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:52 smithi132 bash[18607]: audit 2024-04-25T19:55:51.807477+0000 mon.a (mon.0) 184 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:55:53.276 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:52 smithi132 bash[18607]: audit 2024-04-25T19:55:51.813795+0000 mon.a (mon.0) 185 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:55:53.276 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:52 smithi132 bash[18607]: audit 2024-04-25T19:55:51.813795+0000 mon.a (mon.0) 185 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:55:54.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:53 smithi132 bash[18607]: cluster 2024-04-25T19:55:52.715474+0000 mgr.x (mgr.14152) 80 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:55:54.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:53 smithi132 bash[18607]: cluster 2024-04-25T19:55:52.715474+0000 mgr.x (mgr.14152) 80 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:55:56.155 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:55:56.178 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:55 smithi132 bash[18607]: cluster 2024-04-25T19:55:54.715961+0000 mgr.x (mgr.14152) 81 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:55:56.178 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:55 smithi132 bash[18607]: cluster 2024-04-25T19:55:54.715961+0000 mgr.x (mgr.14152) 81 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:55:58.024 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:57 smithi132 bash[18607]: cluster 2024-04-25T19:55:56.716273+0000 mgr.x (mgr.14152) 82 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:55:58.024 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:57 smithi132 bash[18607]: cluster 2024-04-25T19:55:56.716273+0000 mgr.x (mgr.14152) 82 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:55:59.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:58 smithi132 bash[18607]: audit 2024-04-25T19:55:58.720063+0000 mon.a (mon.0) 186 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T19:55:59.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:58 smithi132 bash[18607]: audit 2024-04-25T19:55:58.720063+0000 mon.a (mon.0) 186 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T19:55:59.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:58 smithi132 bash[18607]: audit 2024-04-25T19:55:58.725709+0000 mon.a (mon.0) 187 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T19:55:59.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:58 smithi132 bash[18607]: audit 2024-04-25T19:55:58.725709+0000 mon.a (mon.0) 187 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T19:55:59.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:58 smithi132 bash[18607]: audit 2024-04-25T19:55:58.727044+0000 mon.a (mon.0) 188 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:55:59.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:58 smithi132 bash[18607]: audit 2024-04-25T19:55:58.727044+0000 mon.a (mon.0) 188 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:56:00.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:59 smithi132 bash[18607]: audit 2024-04-25T19:55:58.716173+0000 mgr.x (mgr.14152) 83 : audit [DBG] from='client.14197 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:56:00.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:59 smithi132 bash[18607]: audit 2024-04-25T19:55:58.716173+0000 mgr.x (mgr.14152) 83 : audit [DBG] from='client.14197 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:56:00.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:59 smithi132 bash[18607]: cluster 2024-04-25T19:55:58.716690+0000 mgr.x (mgr.14152) 84 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:56:00.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:55:59 smithi132 bash[18607]: cluster 2024-04-25T19:55:58.716690+0000 mgr.x (mgr.14152) 84 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:56:02.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:01 smithi132 bash[18607]: cluster 2024-04-25T19:56:00.717099+0000 mgr.x (mgr.14152) 85 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:56:02.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:01 smithi132 bash[18607]: cluster 2024-04-25T19:56:00.717099+0000 mgr.x (mgr.14152) 85 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:56:04.024 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:03 smithi132 bash[18607]: cluster 2024-04-25T19:56:02.717523+0000 mgr.x (mgr.14152) 86 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:56:04.024 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:03 smithi132 bash[18607]: cluster 2024-04-25T19:56:02.717523+0000 mgr.x (mgr.14152) 86 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:56:06.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:05 smithi132 bash[18607]: cluster 2024-04-25T19:56:04.717998+0000 mgr.x (mgr.14152) 87 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:56:06.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:05 smithi132 bash[18607]: cluster 2024-04-25T19:56:04.717998+0000 mgr.x (mgr.14152) 87 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:56:08.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:07 smithi132 bash[18607]: cluster 2024-04-25T19:56:06.718437+0000 mgr.x (mgr.14152) 88 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:56:08.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:07 smithi132 bash[18607]: cluster 2024-04-25T19:56:06.718437+0000 mgr.x (mgr.14152) 88 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:56:08.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:07 smithi132 bash[18607]: audit 2024-04-25T19:56:06.846511+0000 mon.a (mon.0) 189 : audit [INF] from='client.? 172.21.15.132:0/619197340' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8dcbdd31-79f5-43da-9ad5-3594ee6ca003"}]: dispatch 2024-04-25T19:56:08.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:07 smithi132 bash[18607]: audit 2024-04-25T19:56:06.846511+0000 mon.a (mon.0) 189 : audit [INF] from='client.? 172.21.15.132:0/619197340' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8dcbdd31-79f5-43da-9ad5-3594ee6ca003"}]: dispatch 2024-04-25T19:56:08.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:07 smithi132 bash[18607]: audit 2024-04-25T19:56:06.848234+0000 mon.a (mon.0) 190 : audit [INF] from='client.? 172.21.15.132:0/619197340' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8dcbdd31-79f5-43da-9ad5-3594ee6ca003"}]': finished 2024-04-25T19:56:08.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:07 smithi132 bash[18607]: audit 2024-04-25T19:56:06.848234+0000 mon.a (mon.0) 190 : audit [INF] from='client.? 172.21.15.132:0/619197340' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8dcbdd31-79f5-43da-9ad5-3594ee6ca003"}]': finished 2024-04-25T19:56:08.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:07 smithi132 bash[18607]: cluster 2024-04-25T19:56:06.852873+0000 mon.a (mon.0) 191 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-04-25T19:56:08.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:07 smithi132 bash[18607]: cluster 2024-04-25T19:56:06.852873+0000 mon.a (mon.0) 191 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-04-25T19:56:08.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:07 smithi132 bash[18607]: audit 2024-04-25T19:56:06.853190+0000 mon.a (mon.0) 192 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T19:56:08.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:07 smithi132 bash[18607]: audit 2024-04-25T19:56:06.853190+0000 mon.a (mon.0) 192 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T19:56:08.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:07 smithi132 bash[18607]: audit 2024-04-25T19:56:07.540160+0000 mon.a (mon.0) 193 : audit [DBG] from='client.? 172.21.15.132:0/3545192814' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T19:56:08.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:07 smithi132 bash[18607]: audit 2024-04-25T19:56:07.540160+0000 mon.a (mon.0) 193 : audit [DBG] from='client.? 172.21.15.132:0/3545192814' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T19:56:10.225 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:09 smithi132 bash[18607]: cluster 2024-04-25T19:56:08.718875+0000 mgr.x (mgr.14152) 89 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:56:10.225 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:09 smithi132 bash[18607]: cluster 2024-04-25T19:56:08.718875+0000 mgr.x (mgr.14152) 89 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:56:12.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:11 smithi132 bash[18607]: cluster 2024-04-25T19:56:10.719343+0000 mgr.x (mgr.14152) 90 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:56:12.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:11 smithi132 bash[18607]: cluster 2024-04-25T19:56:10.719343+0000 mgr.x (mgr.14152) 90 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:56:14.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:13 smithi132 bash[18607]: cluster 2024-04-25T19:56:12.719818+0000 mgr.x (mgr.14152) 91 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:56:14.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:13 smithi132 bash[18607]: cluster 2024-04-25T19:56:12.719818+0000 mgr.x (mgr.14152) 91 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:56:16.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:15 smithi132 bash[18607]: cluster 2024-04-25T19:56:14.720222+0000 mgr.x (mgr.14152) 92 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:56:16.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:15 smithi132 bash[18607]: cluster 2024-04-25T19:56:14.720222+0000 mgr.x (mgr.14152) 92 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:56:18.184 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:17 smithi132 bash[18607]: cluster 2024-04-25T19:56:16.720545+0000 mgr.x (mgr.14152) 93 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:56:18.184 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:17 smithi132 bash[18607]: cluster 2024-04-25T19:56:16.720545+0000 mgr.x (mgr.14152) 93 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:56:20.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:19 smithi132 bash[18607]: cluster 2024-04-25T19:56:18.720812+0000 mgr.x (mgr.14152) 94 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:56:20.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:19 smithi132 bash[18607]: cluster 2024-04-25T19:56:18.720812+0000 mgr.x (mgr.14152) 94 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:56:20.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:19 smithi132 bash[18607]: audit 2024-04-25T19:56:19.362347+0000 mon.a (mon.0) 194 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-25T19:56:20.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:19 smithi132 bash[18607]: audit 2024-04-25T19:56:19.362347+0000 mon.a (mon.0) 194 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-25T19:56:20.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:19 smithi132 bash[18607]: audit 2024-04-25T19:56:19.364133+0000 mon.a (mon.0) 195 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:56:20.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:19 smithi132 bash[18607]: audit 2024-04-25T19:56:19.364133+0000 mon.a (mon.0) 195 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:56:21.024 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:20 smithi132 bash[18607]: cephadm 2024-04-25T19:56:19.365702+0000 mgr.x (mgr.14152) 95 : cephadm [INF] Deploying daemon osd.1 on smithi132 2024-04-25T19:56:21.024 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:20 smithi132 bash[18607]: cephadm 2024-04-25T19:56:19.365702+0000 mgr.x (mgr.14152) 95 : cephadm [INF] Deploying daemon osd.1 on smithi132 2024-04-25T19:56:22.100 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:21 smithi132 bash[18607]: cluster 2024-04-25T19:56:20.721064+0000 mgr.x (mgr.14152) 96 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:56:22.100 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:21 smithi132 bash[18607]: cluster 2024-04-25T19:56:20.721064+0000 mgr.x (mgr.14152) 96 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:56:23.474 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:23 smithi132 systemd[1]: /etc/systemd/system/ceph-3114c848-033d-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-25T19:56:23.475 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:56:23 smithi132 systemd[1]: /etc/systemd/system/ceph-3114c848-033d-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-25T19:56:23.475 INFO:journalctl@ceph.osd.0.smithi132.stdout:Apr 25 19:56:23 smithi132 systemd[1]: /etc/systemd/system/ceph-3114c848-033d-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-25T19:56:23.759 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:56:23 smithi132 systemd[1]: /etc/systemd/system/ceph-3114c848-033d-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-25T19:56:23.760 INFO:journalctl@ceph.osd.0.smithi132.stdout:Apr 25 19:56:23 smithi132 systemd[1]: /etc/systemd/system/ceph-3114c848-033d-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-25T19:56:23.760 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:23 smithi132 systemd[1]: /etc/systemd/system/ceph-3114c848-033d-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-25T19:56:24.025 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:23 smithi132 bash[18607]: cluster 2024-04-25T19:56:22.721440+0000 mgr.x (mgr.14152) 97 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:56:24.026 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:23 smithi132 bash[18607]: cluster 2024-04-25T19:56:22.721440+0000 mgr.x (mgr.14152) 97 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:56:24.026 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:23 smithi132 bash[18607]: audit 2024-04-25T19:56:23.758348+0000 mon.a (mon.0) 196 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:56:24.026 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:23 smithi132 bash[18607]: audit 2024-04-25T19:56:23.758348+0000 mon.a (mon.0) 196 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:56:24.026 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:23 smithi132 bash[18607]: audit 2024-04-25T19:56:23.767126+0000 mon.a (mon.0) 197 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:56:24.026 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:23 smithi132 bash[18607]: audit 2024-04-25T19:56:23.767126+0000 mon.a (mon.0) 197 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:56:24.026 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:23 smithi132 bash[18607]: audit 2024-04-25T19:56:23.791580+0000 mon.a (mon.0) 198 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:56:24.026 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:23 smithi132 bash[18607]: audit 2024-04-25T19:56:23.791580+0000 mon.a (mon.0) 198 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:56:26.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:25 smithi132 bash[18607]: cluster 2024-04-25T19:56:24.721890+0000 mgr.x (mgr.14152) 98 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:56:26.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:25 smithi132 bash[18607]: cluster 2024-04-25T19:56:24.721890+0000 mgr.x (mgr.14152) 98 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:56:28.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:27 smithi132 bash[18607]: cluster 2024-04-25T19:56:26.722197+0000 mgr.x (mgr.14152) 99 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:56:28.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:27 smithi132 bash[18607]: cluster 2024-04-25T19:56:26.722197+0000 mgr.x (mgr.14152) 99 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:56:30.524 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:30 smithi132 bash[18607]: cluster 2024-04-25T19:56:28.722482+0000 mgr.x (mgr.14152) 100 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:56:30.524 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:30 smithi132 bash[18607]: cluster 2024-04-25T19:56:28.722482+0000 mgr.x (mgr.14152) 100 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:56:30.524 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:30 smithi132 bash[18607]: audit 2024-04-25T19:56:29.027204+0000 mon.a (mon.0) 199 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:56:30.524 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:30 smithi132 bash[18607]: audit 2024-04-25T19:56:29.027204+0000 mon.a (mon.0) 199 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:56:30.524 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:30 smithi132 bash[18607]: audit 2024-04-25T19:56:29.035183+0000 mon.a (mon.0) 200 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:56:30.524 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:30 smithi132 bash[18607]: audit 2024-04-25T19:56:29.035183+0000 mon.a (mon.0) 200 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:56:30.524 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:30 smithi132 bash[18607]: audit 2024-04-25T19:56:29.604992+0000 mon.a (mon.0) 201 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:56:30.524 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:30 smithi132 bash[18607]: audit 2024-04-25T19:56:29.604992+0000 mon.a (mon.0) 201 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:56:30.524 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:30 smithi132 bash[18607]: audit 2024-04-25T19:56:29.605786+0000 mon.a (mon.0) 202 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:56:30.525 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:30 smithi132 bash[18607]: audit 2024-04-25T19:56:29.605786+0000 mon.a (mon.0) 202 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:56:30.525 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:30 smithi132 bash[18607]: audit 2024-04-25T19:56:29.609153+0000 mon.a (mon.0) 203 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:56:30.525 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:30 smithi132 bash[18607]: audit 2024-04-25T19:56:29.609153+0000 mon.a (mon.0) 203 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:56:31.523 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:31 smithi132 bash[18607]: audit 2024-04-25T19:56:30.248360+0000 mon.a (mon.0) 204 : audit [INF] from='osd.1 [v2:172.21.15.132:6810/660596229,v1:172.21.15.132:6811/660596229]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T19:56:31.524 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:31 smithi132 bash[18607]: audit 2024-04-25T19:56:30.248360+0000 mon.a (mon.0) 204 : audit [INF] from='osd.1 [v2:172.21.15.132:6810/660596229,v1:172.21.15.132:6811/660596229]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T19:56:32.378 INFO:teuthology.orchestra.run.smithi132.stdout:Created osd(s) 1 on host 'smithi132' 2024-04-25T19:56:32.397 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:32 smithi132 bash[18607]: cluster 2024-04-25T19:56:30.722899+0000 mgr.x (mgr.14152) 101 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:56:32.397 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:32 smithi132 bash[18607]: cluster 2024-04-25T19:56:30.722899+0000 mgr.x (mgr.14152) 101 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:56:32.397 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:32 smithi132 bash[18607]: audit 2024-04-25T19:56:31.037135+0000 mon.a (mon.0) 205 : audit [INF] from='osd.1 [v2:172.21.15.132:6810/660596229,v1:172.21.15.132:6811/660596229]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-25T19:56:32.397 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:32 smithi132 bash[18607]: audit 2024-04-25T19:56:31.037135+0000 mon.a (mon.0) 205 : audit [INF] from='osd.1 [v2:172.21.15.132:6810/660596229,v1:172.21.15.132:6811/660596229]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-25T19:56:32.397 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:32 smithi132 bash[18607]: cluster 2024-04-25T19:56:31.041505+0000 mon.a (mon.0) 206 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-04-25T19:56:32.397 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:32 smithi132 bash[18607]: cluster 2024-04-25T19:56:31.041505+0000 mon.a (mon.0) 206 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-04-25T19:56:32.397 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:32 smithi132 bash[18607]: audit 2024-04-25T19:56:31.042125+0000 mon.a (mon.0) 207 : audit [INF] from='osd.1 [v2:172.21.15.132:6810/660596229,v1:172.21.15.132:6811/660596229]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-04-25T19:56:32.397 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:32 smithi132 bash[18607]: audit 2024-04-25T19:56:31.042125+0000 mon.a (mon.0) 207 : audit [INF] from='osd.1 [v2:172.21.15.132:6810/660596229,v1:172.21.15.132:6811/660596229]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-04-25T19:56:32.397 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:32 smithi132 bash[18607]: audit 2024-04-25T19:56:31.042661+0000 mon.a (mon.0) 208 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T19:56:32.397 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:32 smithi132 bash[18607]: audit 2024-04-25T19:56:31.042661+0000 mon.a (mon.0) 208 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T19:56:33.035 DEBUG:teuthology.orchestra.run.smithi132:osd.1> sudo journalctl -f -n 0 -u ceph-3114c848-033d-11ef-bc93-c7b262605968@osd.1.service 2024-04-25T19:56:33.038 INFO:tasks.cephadm:Waiting for 2 OSDs to come up... 2024-04-25T19:56:33.038 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-25T19:56:33.524 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:33 smithi132 bash[18607]: audit 2024-04-25T19:56:32.038260+0000 mon.a (mon.0) 209 : audit [INF] from='osd.1 [v2:172.21.15.132:6810/660596229,v1:172.21.15.132:6811/660596229]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-04-25T19:56:33.524 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:33 smithi132 bash[18607]: audit 2024-04-25T19:56:32.038260+0000 mon.a (mon.0) 209 : audit [INF] from='osd.1 [v2:172.21.15.132:6810/660596229,v1:172.21.15.132:6811/660596229]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-04-25T19:56:33.524 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:33 smithi132 bash[18607]: cluster 2024-04-25T19:56:32.044316+0000 mon.a (mon.0) 210 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-04-25T19:56:33.524 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:33 smithi132 bash[18607]: cluster 2024-04-25T19:56:32.044316+0000 mon.a (mon.0) 210 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-04-25T19:56:33.524 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:33 smithi132 bash[18607]: audit 2024-04-25T19:56:32.045179+0000 mon.a (mon.0) 211 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T19:56:33.524 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:33 smithi132 bash[18607]: audit 2024-04-25T19:56:32.045179+0000 mon.a (mon.0) 211 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T19:56:33.524 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:33 smithi132 bash[18607]: audit 2024-04-25T19:56:32.052984+0000 mon.a (mon.0) 212 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T19:56:33.524 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:33 smithi132 bash[18607]: audit 2024-04-25T19:56:32.052984+0000 mon.a (mon.0) 212 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T19:56:33.524 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:33 smithi132 bash[18607]: audit 2024-04-25T19:56:32.347655+0000 mon.a (mon.0) 213 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:56:33.524 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:33 smithi132 bash[18607]: audit 2024-04-25T19:56:32.347655+0000 mon.a (mon.0) 213 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:56:33.525 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:33 smithi132 bash[18607]: audit 2024-04-25T19:56:32.352989+0000 mon.a (mon.0) 214 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:56:33.525 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:33 smithi132 bash[18607]: audit 2024-04-25T19:56:32.352989+0000 mon.a (mon.0) 214 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:56:33.525 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:33 smithi132 bash[18607]: audit 2024-04-25T19:56:32.359455+0000 mon.a (mon.0) 215 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:56:33.525 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:33 smithi132 bash[18607]: audit 2024-04-25T19:56:32.359455+0000 mon.a (mon.0) 215 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:56:34.524 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:34 smithi132 bash[18607]: cluster 2024-04-25T19:56:31.270819+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-25T19:56:34.524 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:34 smithi132 bash[18607]: cluster 2024-04-25T19:56:31.270819+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-25T19:56:34.524 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:34 smithi132 bash[18607]: cluster 2024-04-25T19:56:31.270941+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-25T19:56:34.524 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:34 smithi132 bash[18607]: cluster 2024-04-25T19:56:31.270941+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-25T19:56:34.524 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:34 smithi132 bash[18607]: cluster 2024-04-25T19:56:32.723285+0000 mgr.x (mgr.14152) 102 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:56:34.524 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:34 smithi132 bash[18607]: cluster 2024-04-25T19:56:32.723285+0000 mgr.x (mgr.14152) 102 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T19:56:34.524 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:34 smithi132 bash[18607]: cluster 2024-04-25T19:56:33.051867+0000 mon.a (mon.0) 216 : cluster [INF] osd.1 [v2:172.21.15.132:6810/660596229,v1:172.21.15.132:6811/660596229] boot 2024-04-25T19:56:34.524 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:34 smithi132 bash[18607]: cluster 2024-04-25T19:56:33.051867+0000 mon.a (mon.0) 216 : cluster [INF] osd.1 [v2:172.21.15.132:6810/660596229,v1:172.21.15.132:6811/660596229] boot 2024-04-25T19:56:34.525 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:34 smithi132 bash[18607]: cluster 2024-04-25T19:56:33.051915+0000 mon.a (mon.0) 217 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2024-04-25T19:56:34.525 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:34 smithi132 bash[18607]: cluster 2024-04-25T19:56:33.051915+0000 mon.a (mon.0) 217 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2024-04-25T19:56:34.525 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:34 smithi132 bash[18607]: audit 2024-04-25T19:56:33.052300+0000 mon.a (mon.0) 218 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T19:56:34.525 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:34 smithi132 bash[18607]: audit 2024-04-25T19:56:33.052300+0000 mon.a (mon.0) 218 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T19:56:35.523 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:35 smithi132 bash[18607]: cluster 2024-04-25T19:56:34.064597+0000 mon.a (mon.0) 219 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-04-25T19:56:35.523 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:35 smithi132 bash[18607]: cluster 2024-04-25T19:56:34.064597+0000 mon.a (mon.0) 219 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-04-25T19:56:36.523 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:36 smithi132 bash[18607]: cluster 2024-04-25T19:56:34.723769+0000 mgr.x (mgr.14152) 103 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:56:36.524 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:36 smithi132 bash[18607]: cluster 2024-04-25T19:56:34.723769+0000 mgr.x (mgr.14152) 103 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:56:37.329 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:56:38.318 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:38 smithi132 bash[18607]: cluster 2024-04-25T19:56:36.724279+0000 mgr.x (mgr.14152) 104 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:56:38.319 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:38 smithi132 bash[18607]: cluster 2024-04-25T19:56:36.724279+0000 mgr.x (mgr.14152) 104 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:56:40.523 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:40 smithi132 bash[18607]: cluster 2024-04-25T19:56:38.724827+0000 mgr.x (mgr.14152) 105 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:56:40.523 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:40 smithi132 bash[18607]: cluster 2024-04-25T19:56:38.724827+0000 mgr.x (mgr.14152) 105 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:56:41.099 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T19:56:41.790 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":14,"num_osds":2,"num_up_osds":2,"osd_up_since":1714074993,"num_in_osds":2,"osd_in_since":1714074966,"num_remapped_pgs":0} 2024-04-25T19:56:41.791 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T19:56:42.523 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:42 smithi132 bash[18607]: cluster 2024-04-25T19:56:40.725126+0000 mgr.x (mgr.14152) 106 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:56:42.523 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:42 smithi132 bash[18607]: cluster 2024-04-25T19:56:40.725126+0000 mgr.x (mgr.14152) 106 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:56:42.523 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:42 smithi132 bash[18607]: audit 2024-04-25T19:56:41.100795+0000 mon.a (mon.0) 220 : audit [DBG] from='client.? 172.21.15.132:0/3778061737' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T19:56:42.523 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:42 smithi132 bash[18607]: audit 2024-04-25T19:56:41.100795+0000 mon.a (mon.0) 220 : audit [DBG] from='client.? 172.21.15.132:0/3778061737' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T19:56:43.774 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:43 smithi132 bash[18607]: cephadm 2024-04-25T19:56:42.290658+0000 mgr.x (mgr.14152) 107 : cephadm [INF] Detected new or changed devices on smithi132 2024-04-25T19:56:43.774 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:43 smithi132 bash[18607]: cephadm 2024-04-25T19:56:42.290658+0000 mgr.x (mgr.14152) 107 : cephadm [INF] Detected new or changed devices on smithi132 2024-04-25T19:56:43.774 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:43 smithi132 bash[18607]: audit 2024-04-25T19:56:42.297319+0000 mon.a (mon.0) 221 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:56:43.774 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:43 smithi132 bash[18607]: audit 2024-04-25T19:56:42.297319+0000 mon.a (mon.0) 221 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:56:43.774 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:43 smithi132 bash[18607]: audit 2024-04-25T19:56:42.303903+0000 mon.a (mon.0) 222 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:56:43.774 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:43 smithi132 bash[18607]: audit 2024-04-25T19:56:42.303903+0000 mon.a (mon.0) 222 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:56:43.774 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:43 smithi132 bash[18607]: audit 2024-04-25T19:56:42.305639+0000 mon.a (mon.0) 223 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:56:43.775 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:43 smithi132 bash[18607]: audit 2024-04-25T19:56:42.305639+0000 mon.a (mon.0) 223 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:56:43.775 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:43 smithi132 bash[18607]: audit 2024-04-25T19:56:42.307064+0000 mon.a (mon.0) 224 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:56:43.775 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:43 smithi132 bash[18607]: audit 2024-04-25T19:56:42.307064+0000 mon.a (mon.0) 224 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:56:43.775 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:43 smithi132 bash[18607]: cephadm 2024-04-25T19:56:42.308086+0000 mgr.x (mgr.14152) 108 : cephadm [INF] Adjusting osd_memory_target on smithi132 to 8632M 2024-04-25T19:56:43.775 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:43 smithi132 bash[18607]: cephadm 2024-04-25T19:56:42.308086+0000 mgr.x (mgr.14152) 108 : cephadm [INF] Adjusting osd_memory_target on smithi132 to 8632M 2024-04-25T19:56:43.775 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:43 smithi132 bash[18607]: audit 2024-04-25T19:56:42.312876+0000 mon.a (mon.0) 225 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:56:43.775 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:43 smithi132 bash[18607]: audit 2024-04-25T19:56:42.312876+0000 mon.a (mon.0) 225 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:56:43.775 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:43 smithi132 bash[18607]: audit 2024-04-25T19:56:42.316351+0000 mon.a (mon.0) 226 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:56:43.775 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:43 smithi132 bash[18607]: audit 2024-04-25T19:56:42.316351+0000 mon.a (mon.0) 226 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:56:43.775 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:43 smithi132 bash[18607]: audit 2024-04-25T19:56:42.317626+0000 mon.a (mon.0) 227 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:56:43.775 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:43 smithi132 bash[18607]: audit 2024-04-25T19:56:42.317626+0000 mon.a (mon.0) 227 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:56:43.775 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:43 smithi132 bash[18607]: audit 2024-04-25T19:56:42.324492+0000 mon.a (mon.0) 228 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:56:43.775 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:43 smithi132 bash[18607]: audit 2024-04-25T19:56:42.324492+0000 mon.a (mon.0) 228 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:56:44.773 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:44 smithi132 bash[18607]: cluster 2024-04-25T19:56:42.725498+0000 mgr.x (mgr.14152) 109 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:56:44.773 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:44 smithi132 bash[18607]: cluster 2024-04-25T19:56:42.725498+0000 mgr.x (mgr.14152) 109 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:56:46.517 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:56:46.773 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:46 smithi132 bash[18607]: cluster 2024-04-25T19:56:44.726035+0000 mgr.x (mgr.14152) 110 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:56:46.773 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:46 smithi132 bash[18607]: cluster 2024-04-25T19:56:44.726035+0000 mgr.x (mgr.14152) 110 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:56:47.523 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:47 smithi132 bash[18607]: cluster 2024-04-25T19:56:46.726455+0000 mgr.x (mgr.14152) 111 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:56:47.523 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:47 smithi132 bash[18607]: cluster 2024-04-25T19:56:46.726455+0000 mgr.x (mgr.14152) 111 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:56:49.004 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T19:56:49.004 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":14,"fsid":"3114c848-033d-11ef-bc93-c7b262605968","created":"2024-04-25T19:52:37.558025+0000","modified":"2024-04-25T19:56:34.053507+0000","last_up_change":"2024-04-25T19:56:33.047444+0000","last_in_change":"2024-04-25T19:56:06.847292+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":2,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8345c5ff-03d5-4723-91dd-79e2d3685f55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":3788012097},{"type":"v1","addr":"172.21.15.132:6803","nonce":3788012097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":3788012097},{"type":"v1","addr":"172.21.15.132:6805","nonce":3788012097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":3788012097},{"type":"v1","addr":"172.21.15.132:6809","nonce":3788012097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":3788012097},{"type":"v1","addr":"172.21.15.132:6807","nonce":3788012097}]},"public_addr":"172.21.15.132:6803/3788012097","cluster_addr":"172.21.15.132:6805/3788012097","heartbeat_back_addr":"172.21.15.132:6809/3788012097","heartbeat_front_addr":"172.21.15.132:6807/3788012097","state":["exists","up"]},{"osd":1,"uuid":"8dcbdd31-79f5-43da-9ad5-3594ee6ca003","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":660596229},{"type":"v1","addr":"172.21.15.132:6811","nonce":660596229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":660596229},{"type":"v1","addr":"172.21.15.132:6813","nonce":660596229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":660596229},{"type":"v1","addr":"172.21.15.132:6817","nonce":660596229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":660596229},{"type":"v1","addr":"172.21.15.132:6815","nonce":660596229}]},"public_addr":"172.21.15.132:6811/660596229","cluster_addr":"172.21.15.132:6813/660596229","heartbeat_back_addr":"172.21.15.132:6817/660596229","heartbeat_front_addr":"172.21.15.132:6815/660596229","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:55:40.665761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:56:31.270946+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.132:0/244081902":"2024-04-26T19:53:44.678507+0000","172.21.15.132:6801/3103129571":"2024-04-26T19:53:44.678507+0000","172.21.15.132:6800/3103129571":"2024-04-26T19:53:44.678507+0000","172.21.15.132:6800/926715481":"2024-04-26T19:53:09.950875+0000","172.21.15.132:6801/926715481":"2024-04-26T19:53:09.950875+0000","172.21.15.132:0/475483629":"2024-04-26T19:53:44.678507+0000","172.21.15.132:0/2928415555":"2024-04-26T19:53:09.950875+0000","172.21.15.132:0/1915719166":"2024-04-26T19:53:09.950875+0000","172.21.15.132:0/4217080747":"2024-04-26T19:53:09.950875+0000","172.21.15.132:0/1258487489":"2024-04-26T19:53:44.678507+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:56:49.649 INFO:tasks.cephadm.ceph_manager.ceph:[] 2024-04-25T19:56:49.650 INFO:tasks.cephadm:Setting up client nodes... 2024-04-25T19:56:49.650 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-25T19:56:50.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:49 smithi132 bash[18607]: cluster 2024-04-25T19:56:48.726882+0000 mgr.x (mgr.14152) 112 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:56:50.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:49 smithi132 bash[18607]: cluster 2024-04-25T19:56:48.726882+0000 mgr.x (mgr.14152) 112 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:56:50.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:49 smithi132 bash[18607]: audit 2024-04-25T19:56:49.005378+0000 mon.a (mon.0) 229 : audit [DBG] from='client.? 172.21.15.132:0/2522474328' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T19:56:50.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:49 smithi132 bash[18607]: audit 2024-04-25T19:56:49.005378+0000 mon.a (mon.0) 229 : audit [DBG] from='client.? 172.21.15.132:0/2522474328' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T19:56:52.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:51 smithi132 bash[18607]: cluster 2024-04-25T19:56:50.727339+0000 mgr.x (mgr.14152) 113 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:56:52.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:51 smithi132 bash[18607]: cluster 2024-04-25T19:56:50.727339+0000 mgr.x (mgr.14152) 113 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:56:54.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:53 smithi132 bash[18607]: cluster 2024-04-25T19:56:52.727834+0000 mgr.x (mgr.14152) 114 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:56:54.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:53 smithi132 bash[18607]: cluster 2024-04-25T19:56:52.727834+0000 mgr.x (mgr.14152) 114 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:56:54.384 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:56:56.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:55 smithi132 bash[18607]: cluster 2024-04-25T19:56:54.728274+0000 mgr.x (mgr.14152) 115 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:56:56.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:55 smithi132 bash[18607]: cluster 2024-04-25T19:56:54.728274+0000 mgr.x (mgr.14152) 115 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:56:57.011 INFO:teuthology.orchestra.run.smithi132.stdout:[client.0] 2024-04-25T19:56:57.011 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQCJtSpmFFyqABAA9RWP+X0KdFEytnhPVyCV1Q== 2024-04-25T19:56:57.666 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-04-25T19:56:57.666 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-25T19:56:57.666 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-25T19:56:57.689 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-25T19:56:57.689 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-25T19:56:57.689 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-25T19:56:58.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:57 smithi132 bash[18607]: cluster 2024-04-25T19:56:56.728580+0000 mgr.x (mgr.14152) 116 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:56:58.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:57 smithi132 bash[18607]: cluster 2024-04-25T19:56:56.728580+0000 mgr.x (mgr.14152) 116 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:56:58.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:57 smithi132 bash[18607]: audit 2024-04-25T19:56:57.010843+0000 mon.a (mon.0) 230 : audit [INF] from='client.? 172.21.15.132:0/4289005273' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T19:56:58.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:57 smithi132 bash[18607]: audit 2024-04-25T19:56:57.010843+0000 mon.a (mon.0) 230 : audit [INF] from='client.? 172.21.15.132:0/4289005273' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T19:56:58.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:57 smithi132 bash[18607]: audit 2024-04-25T19:56:57.012563+0000 mon.a (mon.0) 231 : audit [INF] from='client.? 172.21.15.132:0/4289005273' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T19:56:58.024 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:57 smithi132 bash[18607]: audit 2024-04-25T19:56:57.012563+0000 mon.a (mon.0) 231 : audit [INF] from='client.? 172.21.15.132:0/4289005273' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T19:57:00.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:59 smithi132 bash[18607]: cluster 2024-04-25T19:56:58.728926+0000 mgr.x (mgr.14152) 117 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:00.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:56:59 smithi132 bash[18607]: cluster 2024-04-25T19:56:58.728926+0000 mgr.x (mgr.14152) 117 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:02.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:01 smithi132 bash[18607]: cluster 2024-04-25T19:57:00.729341+0000 mgr.x (mgr.14152) 118 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:02.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:01 smithi132 bash[18607]: cluster 2024-04-25T19:57:00.729341+0000 mgr.x (mgr.14152) 118 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:02.458 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:57:04.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:03 smithi132 bash[18607]: cluster 2024-04-25T19:57:02.729844+0000 mgr.x (mgr.14152) 119 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:04.024 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:03 smithi132 bash[18607]: cluster 2024-04-25T19:57:02.729844+0000 mgr.x (mgr.14152) 119 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:05.224 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T19:57:05.939 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":13,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":2833922100},{"type":"v1","addr":"172.21.15.132:6801","nonce":2833922100}]},"active_addr":"172.21.15.132:6801/2833922100","active_change":"2024-04-25T19:53:44.678749+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.132:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.132:0","nonce":1289518386}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.132:0","nonce":2725767790}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.132:0","nonce":4251585279}]}]} 2024-04-25T19:57:05.941 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-25T19:57:05.941 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-25T19:57:05.941 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T19:57:06.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:05 smithi132 bash[18607]: cluster 2024-04-25T19:57:04.730169+0000 mgr.x (mgr.14152) 120 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:06.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:05 smithi132 bash[18607]: cluster 2024-04-25T19:57:04.730169+0000 mgr.x (mgr.14152) 120 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:06.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:05 smithi132 bash[18607]: audit 2024-04-25T19:57:05.220034+0000 mon.a (mon.0) 232 : audit [DBG] from='client.? 172.21.15.132:0/3253084458' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-25T19:57:06.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:05 smithi132 bash[18607]: audit 2024-04-25T19:57:05.220034+0000 mon.a (mon.0) 232 : audit [DBG] from='client.? 172.21.15.132:0/3253084458' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-25T19:57:08.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:07 smithi132 bash[18607]: cluster 2024-04-25T19:57:06.730575+0000 mgr.x (mgr.14152) 121 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:08.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:07 smithi132 bash[18607]: cluster 2024-04-25T19:57:06.730575+0000 mgr.x (mgr.14152) 121 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:10.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:09 smithi132 bash[18607]: cluster 2024-04-25T19:57:08.730924+0000 mgr.x (mgr.14152) 122 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:10.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:09 smithi132 bash[18607]: cluster 2024-04-25T19:57:08.730924+0000 mgr.x (mgr.14152) 122 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:10.674 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:57:12.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:11 smithi132 bash[18607]: cluster 2024-04-25T19:57:10.731312+0000 mgr.x (mgr.14152) 123 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:12.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:11 smithi132 bash[18607]: cluster 2024-04-25T19:57:10.731312+0000 mgr.x (mgr.14152) 123 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:13.335 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T19:57:13.336 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":14,"fsid":"3114c848-033d-11ef-bc93-c7b262605968","created":"2024-04-25T19:52:37.558025+0000","modified":"2024-04-25T19:56:34.053507+0000","last_up_change":"2024-04-25T19:56:33.047444+0000","last_in_change":"2024-04-25T19:56:06.847292+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":2,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8345c5ff-03d5-4723-91dd-79e2d3685f55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":3788012097},{"type":"v1","addr":"172.21.15.132:6803","nonce":3788012097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":3788012097},{"type":"v1","addr":"172.21.15.132:6805","nonce":3788012097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":3788012097},{"type":"v1","addr":"172.21.15.132:6809","nonce":3788012097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":3788012097},{"type":"v1","addr":"172.21.15.132:6807","nonce":3788012097}]},"public_addr":"172.21.15.132:6803/3788012097","cluster_addr":"172.21.15.132:6805/3788012097","heartbeat_back_addr":"172.21.15.132:6809/3788012097","heartbeat_front_addr":"172.21.15.132:6807/3788012097","state":["exists","up"]},{"osd":1,"uuid":"8dcbdd31-79f5-43da-9ad5-3594ee6ca003","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":660596229},{"type":"v1","addr":"172.21.15.132:6811","nonce":660596229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":660596229},{"type":"v1","addr":"172.21.15.132:6813","nonce":660596229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":660596229},{"type":"v1","addr":"172.21.15.132:6817","nonce":660596229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":660596229},{"type":"v1","addr":"172.21.15.132:6815","nonce":660596229}]},"public_addr":"172.21.15.132:6811/660596229","cluster_addr":"172.21.15.132:6813/660596229","heartbeat_back_addr":"172.21.15.132:6817/660596229","heartbeat_front_addr":"172.21.15.132:6815/660596229","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:55:40.665761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:56:31.270946+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.132:0/244081902":"2024-04-26T19:53:44.678507+0000","172.21.15.132:6801/3103129571":"2024-04-26T19:53:44.678507+0000","172.21.15.132:6800/3103129571":"2024-04-26T19:53:44.678507+0000","172.21.15.132:6800/926715481":"2024-04-26T19:53:09.950875+0000","172.21.15.132:6801/926715481":"2024-04-26T19:53:09.950875+0000","172.21.15.132:0/475483629":"2024-04-26T19:53:44.678507+0000","172.21.15.132:0/2928415555":"2024-04-26T19:53:09.950875+0000","172.21.15.132:0/1915719166":"2024-04-26T19:53:09.950875+0000","172.21.15.132:0/4217080747":"2024-04-26T19:53:09.950875+0000","172.21.15.132:0/1258487489":"2024-04-26T19:53:44.678507+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:57:13.978 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-25T19:57:13.982 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T19:57:14.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:13 smithi132 bash[18607]: cluster 2024-04-25T19:57:12.731858+0000 mgr.x (mgr.14152) 124 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:14.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:13 smithi132 bash[18607]: cluster 2024-04-25T19:57:12.731858+0000 mgr.x (mgr.14152) 124 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:14.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:13 smithi132 bash[18607]: audit 2024-04-25T19:57:13.336720+0000 mon.a (mon.0) 233 : audit [DBG] from='client.? 172.21.15.132:0/3737643948' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T19:57:14.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:13 smithi132 bash[18607]: audit 2024-04-25T19:57:13.336720+0000 mon.a (mon.0) 233 : audit [DBG] from='client.? 172.21.15.132:0/3737643948' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T19:57:16.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:15 smithi132 bash[18607]: cluster 2024-04-25T19:57:14.732386+0000 mgr.x (mgr.14152) 125 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:16.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:15 smithi132 bash[18607]: cluster 2024-04-25T19:57:14.732386+0000 mgr.x (mgr.14152) 125 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:18.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:17 smithi132 bash[18607]: cluster 2024-04-25T19:57:16.732735+0000 mgr.x (mgr.14152) 126 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:18.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:17 smithi132 bash[18607]: cluster 2024-04-25T19:57:16.732735+0000 mgr.x (mgr.14152) 126 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:18.699 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:57:20.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:19 smithi132 bash[18607]: cluster 2024-04-25T19:57:18.733146+0000 mgr.x (mgr.14152) 127 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:20.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:19 smithi132 bash[18607]: cluster 2024-04-25T19:57:18.733146+0000 mgr.x (mgr.14152) 127 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:21.232 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T19:57:21.232 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":14,"fsid":"3114c848-033d-11ef-bc93-c7b262605968","created":"2024-04-25T19:52:37.558025+0000","modified":"2024-04-25T19:56:34.053507+0000","last_up_change":"2024-04-25T19:56:33.047444+0000","last_in_change":"2024-04-25T19:56:06.847292+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":2,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8345c5ff-03d5-4723-91dd-79e2d3685f55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":3788012097},{"type":"v1","addr":"172.21.15.132:6803","nonce":3788012097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":3788012097},{"type":"v1","addr":"172.21.15.132:6805","nonce":3788012097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":3788012097},{"type":"v1","addr":"172.21.15.132:6809","nonce":3788012097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":3788012097},{"type":"v1","addr":"172.21.15.132:6807","nonce":3788012097}]},"public_addr":"172.21.15.132:6803/3788012097","cluster_addr":"172.21.15.132:6805/3788012097","heartbeat_back_addr":"172.21.15.132:6809/3788012097","heartbeat_front_addr":"172.21.15.132:6807/3788012097","state":["exists","up"]},{"osd":1,"uuid":"8dcbdd31-79f5-43da-9ad5-3594ee6ca003","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":660596229},{"type":"v1","addr":"172.21.15.132:6811","nonce":660596229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":660596229},{"type":"v1","addr":"172.21.15.132:6813","nonce":660596229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":660596229},{"type":"v1","addr":"172.21.15.132:6817","nonce":660596229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":660596229},{"type":"v1","addr":"172.21.15.132:6815","nonce":660596229}]},"public_addr":"172.21.15.132:6811/660596229","cluster_addr":"172.21.15.132:6813/660596229","heartbeat_back_addr":"172.21.15.132:6817/660596229","heartbeat_front_addr":"172.21.15.132:6815/660596229","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:55:40.665761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:56:31.270946+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.132:0/244081902":"2024-04-26T19:53:44.678507+0000","172.21.15.132:6801/3103129571":"2024-04-26T19:53:44.678507+0000","172.21.15.132:6800/3103129571":"2024-04-26T19:53:44.678507+0000","172.21.15.132:6800/926715481":"2024-04-26T19:53:09.950875+0000","172.21.15.132:6801/926715481":"2024-04-26T19:53:09.950875+0000","172.21.15.132:0/475483629":"2024-04-26T19:53:44.678507+0000","172.21.15.132:0/2928415555":"2024-04-26T19:53:09.950875+0000","172.21.15.132:0/1915719166":"2024-04-26T19:53:09.950875+0000","172.21.15.132:0/4217080747":"2024-04-26T19:53:09.950875+0000","172.21.15.132:0/1258487489":"2024-04-26T19:53:44.678507+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:57:21.872 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-25T19:57:21.873 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-25T19:57:22.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:21 smithi132 bash[18607]: cluster 2024-04-25T19:57:20.733664+0000 mgr.x (mgr.14152) 128 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:22.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:21 smithi132 bash[18607]: cluster 2024-04-25T19:57:20.733664+0000 mgr.x (mgr.14152) 128 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:22.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:21 smithi132 bash[18607]: audit 2024-04-25T19:57:21.233514+0000 mon.a (mon.0) 234 : audit [DBG] from='client.? 172.21.15.132:0/2945691131' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T19:57:22.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:21 smithi132 bash[18607]: audit 2024-04-25T19:57:21.233514+0000 mon.a (mon.0) 234 : audit [DBG] from='client.? 172.21.15.132:0/2945691131' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T19:57:24.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:23 smithi132 bash[18607]: cluster 2024-04-25T19:57:22.734217+0000 mgr.x (mgr.14152) 129 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:24.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:23 smithi132 bash[18607]: cluster 2024-04-25T19:57:22.734217+0000 mgr.x (mgr.14152) 129 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:26.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:25 smithi132 bash[18607]: cluster 2024-04-25T19:57:24.734716+0000 mgr.x (mgr.14152) 130 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:26.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:25 smithi132 bash[18607]: cluster 2024-04-25T19:57:24.734716+0000 mgr.x (mgr.14152) 130 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:26.600 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:57:26.601 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:57:27.896 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:27 smithi132 bash[18607]: cluster 2024-04-25T19:57:26.735215+0000 mgr.x (mgr.14152) 131 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:27.896 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:27 smithi132 bash[18607]: cluster 2024-04-25T19:57:26.735215+0000 mgr.x (mgr.14152) 131 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:29.944 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:29 smithi132 bash[18607]: cluster 2024-04-25T19:57:28.735646+0000 mgr.x (mgr.14152) 132 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:29.944 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:29 smithi132 bash[18607]: cluster 2024-04-25T19:57:28.735646+0000 mgr.x (mgr.14152) 132 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:30.732 INFO:teuthology.orchestra.run.smithi132.stdout:55834574862 2024-04-25T19:57:30.732 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-25T19:57:30.775 INFO:teuthology.orchestra.run.smithi132.stdout:34359738393 2024-04-25T19:57:30.776 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-25T19:57:32.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:31 smithi132 bash[18607]: cluster 2024-04-25T19:57:30.736043+0000 mgr.x (mgr.14152) 133 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:32.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:31 smithi132 bash[18607]: cluster 2024-04-25T19:57:30.736043+0000 mgr.x (mgr.14152) 133 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:34.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:33 smithi132 bash[18607]: cluster 2024-04-25T19:57:32.736501+0000 mgr.x (mgr.14152) 134 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:34.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:33 smithi132 bash[18607]: cluster 2024-04-25T19:57:32.736501+0000 mgr.x (mgr.14152) 134 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:35.453 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:57:35.453 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:57:36.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:35 smithi132 bash[18607]: cluster 2024-04-25T19:57:34.736935+0000 mgr.x (mgr.14152) 135 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:36.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:35 smithi132 bash[18607]: cluster 2024-04-25T19:57:34.736935+0000 mgr.x (mgr.14152) 135 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:38.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:37 smithi132 bash[18607]: cluster 2024-04-25T19:57:36.737236+0000 mgr.x (mgr.14152) 136 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:38.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:37 smithi132 bash[18607]: cluster 2024-04-25T19:57:36.737236+0000 mgr.x (mgr.14152) 136 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:39.135 INFO:teuthology.orchestra.run.smithi132.stdout:34359738395 2024-04-25T19:57:39.143 INFO:teuthology.orchestra.run.smithi132.stdout:55834574863 2024-04-25T19:57:40.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:39 smithi132 bash[18607]: cluster 2024-04-25T19:57:38.737545+0000 mgr.x (mgr.14152) 137 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:40.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:39 smithi132 bash[18607]: cluster 2024-04-25T19:57:38.737545+0000 mgr.x (mgr.14152) 137 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:40.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:39 smithi132 bash[18607]: audit 2024-04-25T19:57:39.137516+0000 mon.a (mon.0) 235 : audit [DBG] from='client.? 172.21.15.132:0/510515656' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T19:57:40.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:39 smithi132 bash[18607]: audit 2024-04-25T19:57:39.137516+0000 mon.a (mon.0) 235 : audit [DBG] from='client.? 172.21.15.132:0/510515656' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T19:57:40.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:39 smithi132 bash[18607]: audit 2024-04-25T19:57:39.145737+0000 mon.a (mon.0) 236 : audit [DBG] from='client.? 172.21.15.132:0/4098018864' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T19:57:40.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:39 smithi132 bash[18607]: audit 2024-04-25T19:57:39.145737+0000 mon.a (mon.0) 236 : audit [DBG] from='client.? 172.21.15.132:0/4098018864' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T19:57:40.217 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738393 got 34359738395 for osd.0 2024-04-25T19:57:40.217 DEBUG:teuthology.parallel:result is None 2024-04-25T19:57:40.232 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574862 got 55834574863 for osd.1 2024-04-25T19:57:40.232 DEBUG:teuthology.parallel:result is None 2024-04-25T19:57:40.232 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-25T19:57:40.232 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T19:57:42.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:41 smithi132 bash[18607]: cluster 2024-04-25T19:57:40.737979+0000 mgr.x (mgr.14152) 138 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:42.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:41 smithi132 bash[18607]: cluster 2024-04-25T19:57:40.737979+0000 mgr.x (mgr.14152) 138 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:43.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:42 smithi132 bash[18607]: audit 2024-04-25T19:57:42.361662+0000 mon.a (mon.0) 237 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:57:43.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:42 smithi132 bash[18607]: audit 2024-04-25T19:57:42.361662+0000 mon.a (mon.0) 237 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:57:44.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:43 smithi132 bash[18607]: cluster 2024-04-25T19:57:42.738438+0000 mgr.x (mgr.14152) 139 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:44.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:43 smithi132 bash[18607]: cluster 2024-04-25T19:57:42.738438+0000 mgr.x (mgr.14152) 139 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:44.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:43 smithi132 bash[18607]: audit 2024-04-25T19:57:42.986530+0000 mon.a (mon.0) 238 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:57:44.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:43 smithi132 bash[18607]: audit 2024-04-25T19:57:42.986530+0000 mon.a (mon.0) 238 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:57:44.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:43 smithi132 bash[18607]: audit 2024-04-25T19:57:42.987318+0000 mon.a (mon.0) 239 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:57:44.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:43 smithi132 bash[18607]: audit 2024-04-25T19:57:42.987318+0000 mon.a (mon.0) 239 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:57:44.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:43 smithi132 bash[18607]: audit 2024-04-25T19:57:42.994601+0000 mon.a (mon.0) 240 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:57:44.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:43 smithi132 bash[18607]: audit 2024-04-25T19:57:42.994601+0000 mon.a (mon.0) 240 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:57:44.998 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:57:46.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:45 smithi132 bash[18607]: cluster 2024-04-25T19:57:44.738917+0000 mgr.x (mgr.14152) 140 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:46.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:45 smithi132 bash[18607]: cluster 2024-04-25T19:57:44.738917+0000 mgr.x (mgr.14152) 140 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:47.517 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T19:57:47.519 INFO:teuthology.orchestra.run.smithi132.stderr:dumped all 2024-04-25T19:57:48.100 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:47 smithi132 bash[18607]: cluster 2024-04-25T19:57:46.739397+0000 mgr.x (mgr.14152) 141 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:48.100 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:47 smithi132 bash[18607]: cluster 2024-04-25T19:57:46.739397+0000 mgr.x (mgr.14152) 141 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:48.101 INFO:teuthology.orchestra.run.smithi132.stdout:{"pg_ready":true,"pg_map":{"version":125,"stamp":"2024-04-25T19:57:46.739193+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":0,"kb":187490304,"kb_used":54280,"kb_used_data":240,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":187436024,"statfs":{"total":191990071296,"available":191934488576,"internally_reserved":0,"allocated":245760,"data_stored":59568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3189,"internal_metadata":55309195},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[],"pool_stats":[],"osd_stats":[{"osd":1,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27140,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93718012,"statfs":{"total":95995035648,"available":95967244288,"internally_reserved":0,"allocated":122880,"data_stored":29784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738396,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27140,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93718012,"statfs":{"total":95995035648,"available":95967244288,"internally_reserved":0,"allocated":122880,"data_stored":29784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-25T19:57:48.101 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T19:57:49.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:48 smithi132 bash[18607]: audit 2024-04-25T19:57:47.518349+0000 mgr.x (mgr.14152) 142 : audit [DBG] from='client.14226 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T19:57:49.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:48 smithi132 bash[18607]: audit 2024-04-25T19:57:47.518349+0000 mgr.x (mgr.14152) 142 : audit [DBG] from='client.14226 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T19:57:50.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:49 smithi132 bash[18607]: cluster 2024-04-25T19:57:48.739875+0000 mgr.x (mgr.14152) 143 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:50.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:49 smithi132 bash[18607]: cluster 2024-04-25T19:57:48.739875+0000 mgr.x (mgr.14152) 143 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:52.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:51 smithi132 bash[18607]: cluster 2024-04-25T19:57:50.740294+0000 mgr.x (mgr.14152) 144 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:52.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:51 smithi132 bash[18607]: cluster 2024-04-25T19:57:50.740294+0000 mgr.x (mgr.14152) 144 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:52.832 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:57:54.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:53 smithi132 bash[18607]: cluster 2024-04-25T19:57:52.740742+0000 mgr.x (mgr.14152) 145 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:54.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:53 smithi132 bash[18607]: cluster 2024-04-25T19:57:52.740742+0000 mgr.x (mgr.14152) 145 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:55.351 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T19:57:55.354 INFO:teuthology.orchestra.run.smithi132.stderr:dumped all 2024-04-25T19:57:55.964 INFO:teuthology.orchestra.run.smithi132.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2024-04-25T19:57:54.741003+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":0,"kb":187490304,"kb_used":54280,"kb_used_data":240,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":187436024,"statfs":{"total":191990071296,"available":191934488576,"internally_reserved":0,"allocated":245760,"data_stored":59568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3189,"internal_metadata":55309195},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[],"pool_stats":[],"osd_stats":[{"osd":1,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27140,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93718012,"statfs":{"total":95995035648,"available":95967244288,"internally_reserved":0,"allocated":122880,"data_stored":29784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738398,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27140,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93718012,"statfs":{"total":95995035648,"available":95967244288,"internally_reserved":0,"allocated":122880,"data_stored":29784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-25T19:57:55.964 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-25T19:57:55.965 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-25T19:57:55.965 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-25T19:57:55.965 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-25T19:57:56.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:55 smithi132 bash[18607]: cluster 2024-04-25T19:57:54.741183+0000 mgr.x (mgr.14152) 146 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:56.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:55 smithi132 bash[18607]: cluster 2024-04-25T19:57:54.741183+0000 mgr.x (mgr.14152) 146 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:57.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:56 smithi132 bash[18607]: audit 2024-04-25T19:57:55.353584+0000 mgr.x (mgr.14152) 147 : audit [DBG] from='client.14228 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T19:57:57.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:56 smithi132 bash[18607]: audit 2024-04-25T19:57:55.353584+0000 mgr.x (mgr.14152) 147 : audit [DBG] from='client.14228 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T19:57:58.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:57 smithi132 bash[18607]: cluster 2024-04-25T19:57:56.741660+0000 mgr.x (mgr.14152) 148 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:57:58.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:57 smithi132 bash[18607]: cluster 2024-04-25T19:57:56.741660+0000 mgr.x (mgr.14152) 148 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:58:00.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:59 smithi132 bash[18607]: cluster 2024-04-25T19:57:58.742158+0000 mgr.x (mgr.14152) 149 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:58:00.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:57:59 smithi132 bash[18607]: cluster 2024-04-25T19:57:58.742158+0000 mgr.x (mgr.14152) 149 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:58:00.702 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:58:02.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:01 smithi132 bash[18607]: cluster 2024-04-25T19:58:00.742523+0000 mgr.x (mgr.14152) 150 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:58:02.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:01 smithi132 bash[18607]: cluster 2024-04-25T19:58:00.742523+0000 mgr.x (mgr.14152) 150 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:58:03.345 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T19:58:03.345 INFO:teuthology.orchestra.run.smithi132.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-25T19:58:03.981 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-25T19:58:03.981 INFO:tasks.cephadm:Setup complete, yielding 2024-04-25T19:58:03.982 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-25T19:58:03.992 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi132.front.sepia.ceph.com 2024-04-25T19:58:03.992 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- bash -c 'ceph fs volume create cephfs' 2024-04-25T19:58:04.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:03 smithi132 bash[18607]: cluster 2024-04-25T19:58:02.742994+0000 mgr.x (mgr.14152) 151 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:58:04.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:03 smithi132 bash[18607]: cluster 2024-04-25T19:58:02.742994+0000 mgr.x (mgr.14152) 151 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:58:04.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:03 smithi132 bash[18607]: audit 2024-04-25T19:58:03.347840+0000 mon.a (mon.0) 241 : audit [DBG] from='client.? 172.21.15.132:0/1711842140' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-25T19:58:04.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:03 smithi132 bash[18607]: audit 2024-04-25T19:58:03.347840+0000 mon.a (mon.0) 241 : audit [DBG] from='client.? 172.21.15.132:0/1711842140' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-25T19:58:06.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:05 smithi132 bash[18607]: cluster 2024-04-25T19:58:04.743502+0000 mgr.x (mgr.14152) 152 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:58:06.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:05 smithi132 bash[18607]: cluster 2024-04-25T19:58:04.743502+0000 mgr.x (mgr.14152) 152 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:58:08.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:07 smithi132 bash[18607]: cluster 2024-04-25T19:58:06.743977+0000 mgr.x (mgr.14152) 153 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:58:08.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:07 smithi132 bash[18607]: cluster 2024-04-25T19:58:06.743977+0000 mgr.x (mgr.14152) 153 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:58:08.717 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:58:10.229 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:09 smithi132 bash[18607]: cluster 2024-04-25T19:58:08.744421+0000 mgr.x (mgr.14152) 154 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:58:10.229 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:09 smithi132 bash[18607]: cluster 2024-04-25T19:58:08.744421+0000 mgr.x (mgr.14152) 154 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:58:12.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:11 smithi132 bash[18607]: cluster 2024-04-25T19:58:10.744910+0000 mgr.x (mgr.14152) 155 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:58:12.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:11 smithi132 bash[18607]: cluster 2024-04-25T19:58:10.744910+0000 mgr.x (mgr.14152) 155 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:58:12.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:11 smithi132 bash[18607]: audit 2024-04-25T19:58:11.296458+0000 mgr.x (mgr.14152) 156 : audit [DBG] from='client.14232 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:58:12.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:11 smithi132 bash[18607]: audit 2024-04-25T19:58:11.296458+0000 mgr.x (mgr.14152) 156 : audit [DBG] from='client.14232 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:58:12.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:11 smithi132 bash[18607]: audit 2024-04-25T19:58:11.297285+0000 mon.a (mon.0) 242 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-04-25T19:58:12.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:11 smithi132 bash[18607]: audit 2024-04-25T19:58:11.297285+0000 mon.a (mon.0) 242 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-04-25T19:58:13.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:12 smithi132 bash[18607]: audit 2024-04-25T19:58:11.880885+0000 mon.a (mon.0) 243 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-04-25T19:58:13.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:12 smithi132 bash[18607]: audit 2024-04-25T19:58:11.880885+0000 mon.a (mon.0) 243 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-04-25T19:58:13.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:12 smithi132 bash[18607]: cluster 2024-04-25T19:58:11.886834+0000 mon.a (mon.0) 244 : cluster [DBG] osdmap e15: 2 total, 2 up, 2 in 2024-04-25T19:58:13.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:12 smithi132 bash[18607]: cluster 2024-04-25T19:58:11.886834+0000 mon.a (mon.0) 244 : cluster [DBG] osdmap e15: 2 total, 2 up, 2 in 2024-04-25T19:58:13.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:12 smithi132 bash[18607]: audit 2024-04-25T19:58:11.888464+0000 mon.a (mon.0) 245 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-04-25T19:58:13.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:12 smithi132 bash[18607]: audit 2024-04-25T19:58:11.888464+0000 mon.a (mon.0) 245 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-04-25T19:58:13.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:12 smithi132 bash[18607]: debug 2024-04-25T19:58:12.924+0000 7fdcfe67f700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T19:58:14.000 INFO:teuthology.run_tasks:Running task cephadm.wait_for_service... 2024-04-25T19:58:14.009 INFO:tasks.cephadm:Waiting for ceph service mds.cephfs to start (timeout 300)... 2024-04-25T19:58:14.009 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T19:58:14.217 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:13 smithi132 bash[18607]: cluster 2024-04-25T19:58:12.745426+0000 mgr.x (mgr.14152) 157 : cluster [DBG] pgmap v139: 32 pgs: 32 unknown; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:58:14.217 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:13 smithi132 bash[18607]: cluster 2024-04-25T19:58:12.745426+0000 mgr.x (mgr.14152) 157 : cluster [DBG] pgmap v139: 32 pgs: 32 unknown; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:58:14.217 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:13 smithi132 bash[18607]: cluster 2024-04-25T19:58:12.880993+0000 mon.a (mon.0) 246 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T19:58:14.217 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:13 smithi132 bash[18607]: cluster 2024-04-25T19:58:12.880993+0000 mon.a (mon.0) 246 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T19:58:14.217 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:13 smithi132 bash[18607]: audit 2024-04-25T19:58:12.891766+0000 mon.a (mon.0) 247 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-04-25T19:58:14.217 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:13 smithi132 bash[18607]: audit 2024-04-25T19:58:12.891766+0000 mon.a (mon.0) 247 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-04-25T19:58:14.217 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:13 smithi132 bash[18607]: cluster 2024-04-25T19:58:12.899730+0000 mon.a (mon.0) 248 : cluster [DBG] osdmap e16: 2 total, 2 up, 2 in 2024-04-25T19:58:14.218 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:13 smithi132 bash[18607]: cluster 2024-04-25T19:58:12.899730+0000 mon.a (mon.0) 248 : cluster [DBG] osdmap e16: 2 total, 2 up, 2 in 2024-04-25T19:58:14.218 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:13 smithi132 bash[18607]: audit 2024-04-25T19:58:12.920847+0000 mon.a (mon.0) 249 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-04-25T19:58:14.218 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:13 smithi132 bash[18607]: audit 2024-04-25T19:58:12.920847+0000 mon.a (mon.0) 249 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-04-25T19:58:14.218 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:13 smithi132 bash[18607]: cluster 2024-04-25T19:58:12.925525+0000 mon.a (mon.0) 250 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T19:58:14.218 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:13 smithi132 bash[18607]: cluster 2024-04-25T19:58:12.925525+0000 mon.a (mon.0) 250 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T19:58:14.218 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:13 smithi132 bash[18607]: cluster 2024-04-25T19:58:12.925842+0000 mon.a (mon.0) 251 : cluster [WRN] Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-25T19:58:14.218 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:13 smithi132 bash[18607]: cluster 2024-04-25T19:58:12.925842+0000 mon.a (mon.0) 251 : cluster [WRN] Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-25T19:58:14.218 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:13 smithi132 bash[18607]: audit 2024-04-25T19:58:12.928002+0000 mon.a (mon.0) 252 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-04-25T19:58:14.218 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:13 smithi132 bash[18607]: audit 2024-04-25T19:58:12.928002+0000 mon.a (mon.0) 252 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-04-25T19:58:14.218 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:13 smithi132 bash[18607]: cluster 2024-04-25T19:58:12.934907+0000 mon.a (mon.0) 253 : cluster [DBG] osdmap e17: 2 total, 2 up, 2 in 2024-04-25T19:58:14.218 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:13 smithi132 bash[18607]: cluster 2024-04-25T19:58:12.934907+0000 mon.a (mon.0) 253 : cluster [DBG] osdmap e17: 2 total, 2 up, 2 in 2024-04-25T19:58:14.218 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:13 smithi132 bash[18607]: cluster 2024-04-25T19:58:12.935369+0000 mon.a (mon.0) 254 : cluster [DBG] fsmap cephfs:0 2024-04-25T19:58:14.218 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:13 smithi132 bash[18607]: cluster 2024-04-25T19:58:12.935369+0000 mon.a (mon.0) 254 : cluster [DBG] fsmap cephfs:0 2024-04-25T19:58:14.219 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:13 smithi132 bash[18607]: cephadm 2024-04-25T19:58:12.940696+0000 mgr.x (mgr.14152) 158 : cephadm [INF] Saving service mds.cephfs spec with placement count:2 2024-04-25T19:58:14.219 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:13 smithi132 bash[18607]: cephadm 2024-04-25T19:58:12.940696+0000 mgr.x (mgr.14152) 158 : cephadm [INF] Saving service mds.cephfs spec with placement count:2 2024-04-25T19:58:14.219 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:13 smithi132 bash[18607]: audit 2024-04-25T19:58:12.948752+0000 mon.a (mon.0) 255 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:58:14.219 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:13 smithi132 bash[18607]: audit 2024-04-25T19:58:12.948752+0000 mon.a (mon.0) 255 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:58:14.219 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:13 smithi132 bash[18607]: audit 2024-04-25T19:58:12.954472+0000 mon.a (mon.0) 256 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:58:14.219 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:13 smithi132 bash[18607]: audit 2024-04-25T19:58:12.954472+0000 mon.a (mon.0) 256 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:58:14.219 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:13 smithi132 bash[18607]: audit 2024-04-25T19:58:12.957500+0000 mon.a (mon.0) 257 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:58:14.219 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:13 smithi132 bash[18607]: audit 2024-04-25T19:58:12.957500+0000 mon.a (mon.0) 257 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:58:14.219 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:13 smithi132 bash[18607]: audit 2024-04-25T19:58:12.958717+0000 mon.a (mon.0) 258 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:58:14.219 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:13 smithi132 bash[18607]: audit 2024-04-25T19:58:12.958717+0000 mon.a (mon.0) 258 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:58:14.219 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:13 smithi132 bash[18607]: audit 2024-04-25T19:58:12.964602+0000 mon.a (mon.0) 259 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:58:14.219 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:13 smithi132 bash[18607]: audit 2024-04-25T19:58:12.964602+0000 mon.a (mon.0) 259 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:58:14.219 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:13 smithi132 bash[18607]: audit 2024-04-25T19:58:12.968275+0000 mon.a (mon.0) 260 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi132.wpuwhp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T19:58:14.220 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:13 smithi132 bash[18607]: audit 2024-04-25T19:58:12.968275+0000 mon.a (mon.0) 260 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi132.wpuwhp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T19:58:14.220 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:13 smithi132 bash[18607]: audit 2024-04-25T19:58:12.969291+0000 mon.a (mon.0) 261 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi132.wpuwhp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T19:58:14.220 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:13 smithi132 bash[18607]: audit 2024-04-25T19:58:12.969291+0000 mon.a (mon.0) 261 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi132.wpuwhp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T19:58:14.220 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:13 smithi132 bash[18607]: audit 2024-04-25T19:58:12.972607+0000 mon.a (mon.0) 262 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:58:14.220 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:13 smithi132 bash[18607]: audit 2024-04-25T19:58:12.972607+0000 mon.a (mon.0) 262 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:58:14.220 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:13 smithi132 bash[18607]: cephadm 2024-04-25T19:58:12.973717+0000 mgr.x (mgr.14152) 159 : cephadm [INF] Deploying daemon mds.cephfs.smithi132.wpuwhp on smithi132 2024-04-25T19:58:14.220 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:13 smithi132 bash[18607]: cephadm 2024-04-25T19:58:12.973717+0000 mgr.x (mgr.14152) 159 : cephadm [INF] Deploying daemon mds.cephfs.smithi132.wpuwhp on smithi132 2024-04-25T19:58:15.248 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:14 smithi132 bash[18607]: cluster 2024-04-25T19:58:13.935599+0000 mon.a (mon.0) 263 : cluster [DBG] osdmap e18: 2 total, 2 up, 2 in 2024-04-25T19:58:15.249 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:14 smithi132 bash[18607]: cluster 2024-04-25T19:58:13.935599+0000 mon.a (mon.0) 263 : cluster [DBG] osdmap e18: 2 total, 2 up, 2 in 2024-04-25T19:58:15.521 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:58:15 smithi132 systemd[1]: /etc/systemd/system/ceph-3114c848-033d-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-25T19:58:15.522 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:58:15 smithi132 systemd[1]: /etc/systemd/system/ceph-3114c848-033d-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-25T19:58:15.522 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:15 smithi132 systemd[1]: /etc/systemd/system/ceph-3114c848-033d-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-25T19:58:15.522 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:15 smithi132 systemd[1]: /etc/systemd/system/ceph-3114c848-033d-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-25T19:58:15.522 INFO:journalctl@ceph.osd.1.smithi132.stdout:Apr 25 19:58:15 smithi132 systemd[1]: /etc/systemd/system/ceph-3114c848-033d-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-25T19:58:15.522 INFO:journalctl@ceph.osd.1.smithi132.stdout:Apr 25 19:58:15 smithi132 systemd[1]: /etc/systemd/system/ceph-3114c848-033d-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-25T19:58:15.523 INFO:journalctl@ceph.osd.0.smithi132.stdout:Apr 25 19:58:15 smithi132 systemd[1]: /etc/systemd/system/ceph-3114c848-033d-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-25T19:58:15.523 INFO:journalctl@ceph.osd.0.smithi132.stdout:Apr 25 19:58:15 smithi132 systemd[1]: /etc/systemd/system/ceph-3114c848-033d-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-25T19:58:16.304 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:15 smithi132 bash[18607]: cluster 2024-04-25T19:58:14.746096+0000 mgr.x (mgr.14152) 160 : cluster [DBG] pgmap v143: 64 pgs: 16 creating+peering, 48 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:58:16.304 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:15 smithi132 bash[18607]: cluster 2024-04-25T19:58:14.746096+0000 mgr.x (mgr.14152) 160 : cluster [DBG] pgmap v143: 64 pgs: 16 creating+peering, 48 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:58:16.304 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:15 smithi132 bash[18607]: cluster 2024-04-25T19:58:14.930316+0000 mon.a (mon.0) 264 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T19:58:16.304 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:15 smithi132 bash[18607]: cluster 2024-04-25T19:58:14.930316+0000 mon.a (mon.0) 264 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T19:58:16.304 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:15 smithi132 bash[18607]: cluster 2024-04-25T19:58:14.942749+0000 mon.a (mon.0) 265 : cluster [DBG] osdmap e19: 2 total, 2 up, 2 in 2024-04-25T19:58:16.304 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:15 smithi132 bash[18607]: cluster 2024-04-25T19:58:14.942749+0000 mon.a (mon.0) 265 : cluster [DBG] osdmap e19: 2 total, 2 up, 2 in 2024-04-25T19:58:16.304 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:15 smithi132 bash[18607]: audit 2024-04-25T19:58:15.634624+0000 mon.a (mon.0) 266 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:58:16.304 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:15 smithi132 bash[18607]: audit 2024-04-25T19:58:15.634624+0000 mon.a (mon.0) 266 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:58:16.304 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:15 smithi132 bash[18607]: audit 2024-04-25T19:58:15.640481+0000 mon.a (mon.0) 267 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:58:16.304 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:15 smithi132 bash[18607]: audit 2024-04-25T19:58:15.640481+0000 mon.a (mon.0) 267 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:58:16.304 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:15 smithi132 bash[18607]: audit 2024-04-25T19:58:15.650575+0000 mon.a (mon.0) 268 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:58:16.305 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:15 smithi132 bash[18607]: audit 2024-04-25T19:58:15.650575+0000 mon.a (mon.0) 268 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:58:16.305 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:15 smithi132 bash[18607]: audit 2024-04-25T19:58:15.651157+0000 mon.a (mon.0) 269 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi132.wyahux", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T19:58:16.305 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:15 smithi132 bash[18607]: audit 2024-04-25T19:58:15.651157+0000 mon.a (mon.0) 269 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi132.wyahux", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T19:58:16.305 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:15 smithi132 bash[18607]: audit 2024-04-25T19:58:15.654013+0000 mon.a (mon.0) 270 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi132.wyahux", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T19:58:16.305 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:15 smithi132 bash[18607]: audit 2024-04-25T19:58:15.654013+0000 mon.a (mon.0) 270 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi132.wyahux", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T19:58:16.305 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:15 smithi132 bash[18607]: audit 2024-04-25T19:58:15.655731+0000 mon.a (mon.0) 271 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:58:16.305 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:15 smithi132 bash[18607]: audit 2024-04-25T19:58:15.655731+0000 mon.a (mon.0) 271 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:58:17.226 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:16 smithi132 bash[18607]: cephadm 2024-04-25T19:58:15.656534+0000 mgr.x (mgr.14152) 161 : cephadm [INF] Deploying daemon mds.cephfs.smithi132.wyahux on smithi132 2024-04-25T19:58:17.226 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:16 smithi132 bash[18607]: cephadm 2024-04-25T19:58:15.656534+0000 mgr.x (mgr.14152) 161 : cephadm [INF] Deploying daemon mds.cephfs.smithi132.wyahux on smithi132 2024-04-25T19:58:18.242 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:58:18 smithi132 systemd[1]: /etc/systemd/system/ceph-3114c848-033d-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-25T19:58:18.243 INFO:journalctl@ceph.osd.0.smithi132.stdout:Apr 25 19:58:18 smithi132 systemd[1]: /etc/systemd/system/ceph-3114c848-033d-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-25T19:58:18.243 INFO:journalctl@ceph.osd.1.smithi132.stdout:Apr 25 19:58:18 smithi132 systemd[1]: /etc/systemd/system/ceph-3114c848-033d-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-25T19:58:18.243 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:17 smithi132 bash[18607]: cluster 2024-04-25T19:58:16.746510+0000 mgr.x (mgr.14152) 162 : cluster [DBG] pgmap v145: 64 pgs: 16 creating+peering, 48 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:58:18.243 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:17 smithi132 bash[18607]: cluster 2024-04-25T19:58:16.746510+0000 mgr.x (mgr.14152) 162 : cluster [DBG] pgmap v145: 64 pgs: 16 creating+peering, 48 active+undersized; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T19:58:18.243 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:18 smithi132 systemd[1]: /etc/systemd/system/ceph-3114c848-033d-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-25T19:58:18.521 INFO:journalctl@ceph.osd.0.smithi132.stdout:Apr 25 19:58:18 smithi132 systemd[1]: /etc/systemd/system/ceph-3114c848-033d-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-25T19:58:18.522 INFO:journalctl@ceph.osd.1.smithi132.stdout:Apr 25 19:58:18 smithi132 systemd[1]: /etc/systemd/system/ceph-3114c848-033d-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-25T19:58:18.522 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:58:18 smithi132 systemd[1]: /etc/systemd/system/ceph-3114c848-033d-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-25T19:58:18.522 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:18 smithi132 systemd[1]: /etc/systemd/system/ceph-3114c848-033d-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-25T19:58:18.727 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:58:19.017 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:18 smithi132 bash[18607]: cluster 2024-04-25T19:58:17.950351+0000 mon.a (mon.0) 272 : cluster [DBG] mds.? [v2:172.21.15.132:6818/3854074450,v1:172.21.15.132:6819/3854074450] up:boot 2024-04-25T19:58:19.017 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:18 smithi132 bash[18607]: cluster 2024-04-25T19:58:17.950351+0000 mon.a (mon.0) 272 : cluster [DBG] mds.? [v2:172.21.15.132:6818/3854074450,v1:172.21.15.132:6819/3854074450] up:boot 2024-04-25T19:58:19.017 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:18 smithi132 bash[18607]: cluster 2024-04-25T19:58:17.950418+0000 mon.a (mon.0) 273 : cluster [INF] daemon mds.cephfs.smithi132.wpuwhp assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-04-25T19:58:19.017 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:19 smithi132 bash[18607]: cluster 2024-04-25T19:58:17.950418+0000 mon.a (mon.0) 273 : cluster [INF] daemon mds.cephfs.smithi132.wpuwhp assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-04-25T19:58:19.017 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:19 smithi132 bash[18607]: cluster 2024-04-25T19:58:17.950489+0000 mon.a (mon.0) 274 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-25T19:58:19.017 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:19 smithi132 bash[18607]: cluster 2024-04-25T19:58:17.950489+0000 mon.a (mon.0) 274 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-25T19:58:19.017 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:19 smithi132 bash[18607]: cluster 2024-04-25T19:58:17.950537+0000 mon.a (mon.0) 275 : cluster [INF] Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-25T19:58:19.017 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:19 smithi132 bash[18607]: cluster 2024-04-25T19:58:17.950537+0000 mon.a (mon.0) 275 : cluster [INF] Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-25T19:58:19.017 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:19 smithi132 bash[18607]: cluster 2024-04-25T19:58:17.950589+0000 mon.a (mon.0) 276 : cluster [INF] Cluster is now healthy 2024-04-25T19:58:19.017 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:19 smithi132 bash[18607]: cluster 2024-04-25T19:58:17.950589+0000 mon.a (mon.0) 276 : cluster [INF] Cluster is now healthy 2024-04-25T19:58:19.017 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:19 smithi132 bash[18607]: cluster 2024-04-25T19:58:17.950871+0000 mon.a (mon.0) 277 : cluster [DBG] fsmap cephfs:0 1 up:standby 2024-04-25T19:58:19.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:19 smithi132 bash[18607]: cluster 2024-04-25T19:58:17.950871+0000 mon.a (mon.0) 277 : cluster [DBG] fsmap cephfs:0 1 up:standby 2024-04-25T19:58:19.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:19 smithi132 bash[18607]: cluster 2024-04-25T19:58:17.955588+0000 mon.a (mon.0) 278 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi132.wpuwhp=up:creating} 2024-04-25T19:58:19.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:19 smithi132 bash[18607]: cluster 2024-04-25T19:58:17.955588+0000 mon.a (mon.0) 278 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi132.wpuwhp=up:creating} 2024-04-25T19:58:19.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:19 smithi132 bash[18607]: audit 2024-04-25T19:58:17.956179+0000 mon.a (mon.0) 279 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi132.wpuwhp"}]: dispatch 2024-04-25T19:58:19.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:19 smithi132 bash[18607]: audit 2024-04-25T19:58:17.956179+0000 mon.a (mon.0) 279 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi132.wpuwhp"}]: dispatch 2024-04-25T19:58:19.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:19 smithi132 bash[18607]: cluster 2024-04-25T19:58:17.981635+0000 mon.a (mon.0) 280 : cluster [INF] daemon mds.cephfs.smithi132.wpuwhp is now active in filesystem cephfs as rank 0 2024-04-25T19:58:19.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:19 smithi132 bash[18607]: cluster 2024-04-25T19:58:17.981635+0000 mon.a (mon.0) 280 : cluster [INF] daemon mds.cephfs.smithi132.wpuwhp is now active in filesystem cephfs as rank 0 2024-04-25T19:58:19.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:19 smithi132 bash[18607]: audit 2024-04-25T19:58:18.402445+0000 mon.a (mon.0) 281 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:58:19.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:19 smithi132 bash[18607]: audit 2024-04-25T19:58:18.402445+0000 mon.a (mon.0) 281 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:58:19.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:19 smithi132 bash[18607]: audit 2024-04-25T19:58:18.410446+0000 mon.a (mon.0) 282 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:58:19.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:19 smithi132 bash[18607]: audit 2024-04-25T19:58:18.410446+0000 mon.a (mon.0) 282 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:58:19.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:19 smithi132 bash[18607]: audit 2024-04-25T19:58:18.428553+0000 mon.a (mon.0) 283 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:58:19.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:19 smithi132 bash[18607]: audit 2024-04-25T19:58:18.428553+0000 mon.a (mon.0) 283 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:58:19.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:19 smithi132 bash[18607]: audit 2024-04-25T19:58:18.430948+0000 mon.a (mon.0) 284 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:58:19.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:19 smithi132 bash[18607]: audit 2024-04-25T19:58:18.430948+0000 mon.a (mon.0) 284 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:58:19.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:19 smithi132 bash[18607]: audit 2024-04-25T19:58:18.434359+0000 mon.a (mon.0) 285 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:58:19.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:19 smithi132 bash[18607]: audit 2024-04-25T19:58:18.434359+0000 mon.a (mon.0) 285 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:58:19.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:19 smithi132 bash[18607]: audit 2024-04-25T19:58:18.451261+0000 mon.a (mon.0) 286 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:58:19.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:19 smithi132 bash[18607]: audit 2024-04-25T19:58:18.451261+0000 mon.a (mon.0) 286 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:58:20.238 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:19 smithi132 bash[18607]: cluster 2024-04-25T19:58:18.747143+0000 mgr.x (mgr.14152) 163 : cluster [DBG] pgmap v146: 64 pgs: 3 active+undersized+degraded, 61 active+undersized; 582 B data, 57 MiB used, 179 GiB / 179 GiB avail; 7/21 objects degraded (33.333%) 2024-04-25T19:58:20.238 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:19 smithi132 bash[18607]: cluster 2024-04-25T19:58:18.747143+0000 mgr.x (mgr.14152) 163 : cluster [DBG] pgmap v146: 64 pgs: 3 active+undersized+degraded, 61 active+undersized; 582 B data, 57 MiB used, 179 GiB / 179 GiB avail; 7/21 objects degraded (33.333%) 2024-04-25T19:58:20.238 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:19 smithi132 bash[18607]: cluster 2024-04-25T19:58:18.964172+0000 mon.a (mon.0) 287 : cluster [DBG] mds.? [v2:172.21.15.132:6818/3854074450,v1:172.21.15.132:6819/3854074450] up:active 2024-04-25T19:58:20.238 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:19 smithi132 bash[18607]: cluster 2024-04-25T19:58:18.964172+0000 mon.a (mon.0) 287 : cluster [DBG] mds.? [v2:172.21.15.132:6818/3854074450,v1:172.21.15.132:6819/3854074450] up:active 2024-04-25T19:58:20.238 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:19 smithi132 bash[18607]: cluster 2024-04-25T19:58:18.964269+0000 mon.a (mon.0) 288 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi132.wpuwhp=up:active} 2024-04-25T19:58:20.238 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:19 smithi132 bash[18607]: cluster 2024-04-25T19:58:18.964269+0000 mon.a (mon.0) 288 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi132.wpuwhp=up:active} 2024-04-25T19:58:20.238 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:19 smithi132 bash[18607]: cluster 2024-04-25T19:58:19.433290+0000 mon.a (mon.0) 289 : cluster [WRN] Health check failed: Degraded data redundancy: 7/21 objects degraded (33.333%), 3 pgs degraded (PG_DEGRADED) 2024-04-25T19:58:20.238 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:19 smithi132 bash[18607]: cluster 2024-04-25T19:58:19.433290+0000 mon.a (mon.0) 289 : cluster [WRN] Health check failed: Degraded data redundancy: 7/21 objects degraded (33.333%), 3 pgs degraded (PG_DEGRADED) 2024-04-25T19:58:20.238 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:19 smithi132 bash[18607]: audit 2024-04-25T19:58:19.777045+0000 mon.a (mon.0) 290 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:58:20.238 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:19 smithi132 bash[18607]: audit 2024-04-25T19:58:19.777045+0000 mon.a (mon.0) 290 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:58:21.704 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T19:58:21.704 INFO:teuthology.orchestra.run.smithi132.stdout:[{"events": ["2024-04-25T19:58:18.434581Z service:mds.cephfs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "running": 0, "size": 2}}, {"events": ["2024-04-25T19:54:53.573757Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:56:29.019130Z", "running": 1, "size": 1}}, {"events": ["2024-04-25T19:54:28.767400Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:56:29.018992Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:56:29.019246Z", "running": 2, "size": 2}, "unmanaged": true}] 2024-04-25T19:58:21.969 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:21 smithi132 bash[18607]: cluster 2024-04-25T19:58:20.747987+0000 mgr.x (mgr.14152) 164 : cluster [DBG] pgmap v147: 64 pgs: 10 active+undersized+degraded, 54 active+undersized; 2.4 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 6.5 KiB/s wr, 12 op/s; 21/63 objects degraded (33.333%) 2024-04-25T19:58:21.969 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:21 smithi132 bash[18607]: cluster 2024-04-25T19:58:20.747987+0000 mgr.x (mgr.14152) 164 : cluster [DBG] pgmap v147: 64 pgs: 10 active+undersized+degraded, 54 active+undersized; 2.4 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 6.5 KiB/s wr, 12 op/s; 21/63 objects degraded (33.333%) 2024-04-25T19:58:21.969 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:21 smithi132 bash[18607]: cluster 2024-04-25T19:58:20.970765+0000 mon.a (mon.0) 291 : cluster [DBG] mds.? [v2:172.21.15.132:6820/2140778992,v1:172.21.15.132:6821/2140778992] up:boot 2024-04-25T19:58:21.969 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:21 smithi132 bash[18607]: cluster 2024-04-25T19:58:20.970765+0000 mon.a (mon.0) 291 : cluster [DBG] mds.? [v2:172.21.15.132:6820/2140778992,v1:172.21.15.132:6821/2140778992] up:boot 2024-04-25T19:58:21.969 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:21 smithi132 bash[18607]: cluster 2024-04-25T19:58:20.970855+0000 mon.a (mon.0) 292 : cluster [INF] Dropping low affinity active daemon mds.cephfs.smithi132.wpuwhp in favor of higher affinity standby. 2024-04-25T19:58:21.970 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:21 smithi132 bash[18607]: cluster 2024-04-25T19:58:20.970855+0000 mon.a (mon.0) 292 : cluster [INF] Dropping low affinity active daemon mds.cephfs.smithi132.wpuwhp in favor of higher affinity standby. 2024-04-25T19:58:21.970 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:21 smithi132 bash[18607]: cluster 2024-04-25T19:58:20.970903+0000 mon.a (mon.0) 293 : cluster [WRN] Replacing daemon mds.cephfs.smithi132.wpuwhp as rank 0 with standby daemon mds.cephfs.smithi132.wyahux 2024-04-25T19:58:21.970 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:21 smithi132 bash[18607]: cluster 2024-04-25T19:58:20.970903+0000 mon.a (mon.0) 293 : cluster [WRN] Replacing daemon mds.cephfs.smithi132.wpuwhp as rank 0 with standby daemon mds.cephfs.smithi132.wyahux 2024-04-25T19:58:21.970 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:21 smithi132 bash[18607]: cluster 2024-04-25T19:58:20.971404+0000 mon.a (mon.0) 294 : cluster [WRN] Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T19:58:21.970 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:21 smithi132 bash[18607]: cluster 2024-04-25T19:58:20.971404+0000 mon.a (mon.0) 294 : cluster [WRN] Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T19:58:21.970 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:21 smithi132 bash[18607]: cluster 2024-04-25T19:58:20.971451+0000 mon.a (mon.0) 295 : cluster [WRN] Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-25T19:58:21.970 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:21 smithi132 bash[18607]: cluster 2024-04-25T19:58:20.971451+0000 mon.a (mon.0) 295 : cluster [WRN] Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-25T19:58:21.970 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:21 smithi132 bash[18607]: cluster 2024-04-25T19:58:20.971838+0000 mon.a (mon.0) 296 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi132.wpuwhp=up:active} 1 up:standby 2024-04-25T19:58:21.970 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:21 smithi132 bash[18607]: cluster 2024-04-25T19:58:20.971838+0000 mon.a (mon.0) 296 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi132.wpuwhp=up:active} 1 up:standby 2024-04-25T19:58:21.970 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:21 smithi132 bash[18607]: cluster 2024-04-25T19:58:20.978816+0000 mon.a (mon.0) 297 : cluster [DBG] osdmap e20: 2 total, 2 up, 2 in 2024-04-25T19:58:21.970 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:21 smithi132 bash[18607]: cluster 2024-04-25T19:58:20.978816+0000 mon.a (mon.0) 297 : cluster [DBG] osdmap e20: 2 total, 2 up, 2 in 2024-04-25T19:58:21.970 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:21 smithi132 bash[18607]: cluster 2024-04-25T19:58:20.980281+0000 mon.a (mon.0) 298 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi132.wyahux=up:replay} 2024-04-25T19:58:21.971 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:21 smithi132 bash[18607]: cluster 2024-04-25T19:58:20.980281+0000 mon.a (mon.0) 298 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi132.wyahux=up:replay} 2024-04-25T19:58:21.971 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:21 smithi132 bash[18607]: audit 2024-04-25T19:58:20.981174+0000 mon.a (mon.0) 299 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi132.wyahux"}]: dispatch 2024-04-25T19:58:21.971 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:21 smithi132 bash[18607]: audit 2024-04-25T19:58:20.981174+0000 mon.a (mon.0) 299 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi132.wyahux"}]: dispatch 2024-04-25T19:58:22.319 INFO:tasks.cephadm:mds.cephfs has 0/2 2024-04-25T19:58:23.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:22 smithi132 bash[18607]: audit 2024-04-25T19:58:21.705115+0000 mgr.x (mgr.14152) 165 : audit [DBG] from='client.14240 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T19:58:23.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:22 smithi132 bash[18607]: audit 2024-04-25T19:58:21.705115+0000 mgr.x (mgr.14152) 165 : audit [DBG] from='client.14240 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T19:58:23.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:22 smithi132 bash[18607]: cluster 2024-04-25T19:58:21.975817+0000 mon.a (mon.0) 300 : cluster [INF] Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-25T19:58:23.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:22 smithi132 bash[18607]: cluster 2024-04-25T19:58:21.975817+0000 mon.a (mon.0) 300 : cluster [INF] Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-25T19:58:23.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:22 smithi132 bash[18607]: cluster 2024-04-25T19:58:21.982094+0000 mon.a (mon.0) 301 : cluster [DBG] mds.? [v2:172.21.15.132:6820/2140778992,v1:172.21.15.132:6821/2140778992] up:reconnect 2024-04-25T19:58:23.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:22 smithi132 bash[18607]: cluster 2024-04-25T19:58:21.982094+0000 mon.a (mon.0) 301 : cluster [DBG] mds.? [v2:172.21.15.132:6820/2140778992,v1:172.21.15.132:6821/2140778992] up:reconnect 2024-04-25T19:58:23.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:22 smithi132 bash[18607]: cluster 2024-04-25T19:58:21.982202+0000 mon.a (mon.0) 302 : cluster [DBG] mds.? [v2:172.21.15.132:6818/3924249517,v1:172.21.15.132:6819/3924249517] up:boot 2024-04-25T19:58:23.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:22 smithi132 bash[18607]: cluster 2024-04-25T19:58:21.982202+0000 mon.a (mon.0) 302 : cluster [DBG] mds.? [v2:172.21.15.132:6818/3924249517,v1:172.21.15.132:6819/3924249517] up:boot 2024-04-25T19:58:23.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:22 smithi132 bash[18607]: cluster 2024-04-25T19:58:21.982343+0000 mon.a (mon.0) 303 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi132.wyahux=up:reconnect} 1 up:standby 2024-04-25T19:58:23.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:22 smithi132 bash[18607]: cluster 2024-04-25T19:58:21.982343+0000 mon.a (mon.0) 303 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi132.wyahux=up:reconnect} 1 up:standby 2024-04-25T19:58:23.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:22 smithi132 bash[18607]: audit 2024-04-25T19:58:21.983188+0000 mon.a (mon.0) 304 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi132.wpuwhp"}]: dispatch 2024-04-25T19:58:23.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:22 smithi132 bash[18607]: audit 2024-04-25T19:58:21.983188+0000 mon.a (mon.0) 304 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi132.wpuwhp"}]: dispatch 2024-04-25T19:58:23.320 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T19:58:24.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:23 smithi132 bash[18607]: cluster 2024-04-25T19:58:22.748717+0000 mgr.x (mgr.14152) 166 : cluster [DBG] pgmap v149: 64 pgs: 10 active+undersized+degraded, 54 active+undersized; 2.4 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 3.2 KiB/s wr, 6 op/s; 21/63 objects degraded (33.333%) 2024-04-25T19:58:24.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:23 smithi132 bash[18607]: cluster 2024-04-25T19:58:22.748717+0000 mgr.x (mgr.14152) 166 : cluster [DBG] pgmap v149: 64 pgs: 10 active+undersized+degraded, 54 active+undersized; 2.4 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 3.2 KiB/s wr, 6 op/s; 21/63 objects degraded (33.333%) 2024-04-25T19:58:24.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:23 smithi132 bash[18607]: cluster 2024-04-25T19:58:22.988884+0000 mon.a (mon.0) 305 : cluster [DBG] mds.? [v2:172.21.15.132:6820/2140778992,v1:172.21.15.132:6821/2140778992] up:rejoin 2024-04-25T19:58:24.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:23 smithi132 bash[18607]: cluster 2024-04-25T19:58:22.988884+0000 mon.a (mon.0) 305 : cluster [DBG] mds.? [v2:172.21.15.132:6820/2140778992,v1:172.21.15.132:6821/2140778992] up:rejoin 2024-04-25T19:58:24.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:23 smithi132 bash[18607]: cluster 2024-04-25T19:58:22.988981+0000 mon.a (mon.0) 306 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi132.wyahux=up:rejoin} 1 up:standby 2024-04-25T19:58:24.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:23 smithi132 bash[18607]: cluster 2024-04-25T19:58:22.988981+0000 mon.a (mon.0) 306 : cluster [DBG] fsmap cephfs:1/1 {0=cephfs.smithi132.wyahux=up:rejoin} 1 up:standby 2024-04-25T19:58:24.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:23 smithi132 bash[18607]: cluster 2024-04-25T19:58:22.989256+0000 mon.a (mon.0) 307 : cluster [INF] daemon mds.cephfs.smithi132.wyahux is now active in filesystem cephfs as rank 0 2024-04-25T19:58:24.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:23 smithi132 bash[18607]: cluster 2024-04-25T19:58:22.989256+0000 mon.a (mon.0) 307 : cluster [INF] daemon mds.cephfs.smithi132.wyahux is now active in filesystem cephfs as rank 0 2024-04-25T19:58:25.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:24 smithi132 bash[18607]: cluster 2024-04-25T19:58:23.985483+0000 mon.a (mon.0) 308 : cluster [INF] Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T19:58:25.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:24 smithi132 bash[18607]: cluster 2024-04-25T19:58:23.985483+0000 mon.a (mon.0) 308 : cluster [INF] Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T19:58:25.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:24 smithi132 bash[18607]: cluster 2024-04-25T19:58:23.991923+0000 mon.a (mon.0) 309 : cluster [DBG] mds.? [v2:172.21.15.132:6820/2140778992,v1:172.21.15.132:6821/2140778992] up:active 2024-04-25T19:58:25.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:24 smithi132 bash[18607]: cluster 2024-04-25T19:58:23.991923+0000 mon.a (mon.0) 309 : cluster [DBG] mds.? [v2:172.21.15.132:6820/2140778992,v1:172.21.15.132:6821/2140778992] up:active 2024-04-25T19:58:25.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:24 smithi132 bash[18607]: cluster 2024-04-25T19:58:23.991989+0000 mon.a (mon.0) 310 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi132.wyahux=up:active} 1 up:standby 2024-04-25T19:58:25.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:24 smithi132 bash[18607]: cluster 2024-04-25T19:58:23.991989+0000 mon.a (mon.0) 310 : cluster [DBG] fsmap cephfs:1 {0=cephfs.smithi132.wyahux=up:active} 1 up:standby 2024-04-25T19:58:25.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:24 smithi132 bash[18607]: audit 2024-04-25T19:58:24.136069+0000 mon.a (mon.0) 311 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:58:25.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:24 smithi132 bash[18607]: audit 2024-04-25T19:58:24.136069+0000 mon.a (mon.0) 311 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:58:25.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:24 smithi132 bash[18607]: audit 2024-04-25T19:58:24.139277+0000 mon.a (mon.0) 312 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:58:25.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:24 smithi132 bash[18607]: audit 2024-04-25T19:58:24.139277+0000 mon.a (mon.0) 312 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:58:25.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:24 smithi132 bash[18607]: audit 2024-04-25T19:58:24.140477+0000 mon.a (mon.0) 313 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:58:25.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:24 smithi132 bash[18607]: audit 2024-04-25T19:58:24.140477+0000 mon.a (mon.0) 313 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:58:25.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:24 smithi132 bash[18607]: audit 2024-04-25T19:58:24.141302+0000 mon.a (mon.0) 314 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:58:25.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:24 smithi132 bash[18607]: audit 2024-04-25T19:58:24.141302+0000 mon.a (mon.0) 314 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:58:25.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:24 smithi132 bash[18607]: audit 2024-04-25T19:58:24.144451+0000 mon.a (mon.0) 315 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:58:25.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:24 smithi132 bash[18607]: audit 2024-04-25T19:58:24.144451+0000 mon.a (mon.0) 315 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:58:25.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:24 smithi132 bash[18607]: audit 2024-04-25T19:58:24.163509+0000 mon.a (mon.0) 316 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:58:25.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:24 smithi132 bash[18607]: audit 2024-04-25T19:58:24.163509+0000 mon.a (mon.0) 316 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:58:26.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:25 smithi132 bash[18607]: cluster 2024-04-25T19:58:24.749751+0000 mgr.x (mgr.14152) 167 : cluster [DBG] pgmap v150: 64 pgs: 13 active+undersized+degraded, 51 active+undersized; 2.4 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 8.2 KiB/s rd, 2.2 KiB/s wr, 11 op/s; 22/66 objects degraded (33.333%) 2024-04-25T19:58:26.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:25 smithi132 bash[18607]: cluster 2024-04-25T19:58:24.749751+0000 mgr.x (mgr.14152) 167 : cluster [DBG] pgmap v150: 64 pgs: 13 active+undersized+degraded, 51 active+undersized; 2.4 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 8.2 KiB/s rd, 2.2 KiB/s wr, 11 op/s; 22/66 objects degraded (33.333%) 2024-04-25T19:58:26.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:25 smithi132 bash[18607]: cluster 2024-04-25T19:58:25.143379+0000 mon.a (mon.0) 317 : cluster [WRN] Health check update: Degraded data redundancy: 22/66 objects degraded (33.333%), 13 pgs degraded (PG_DEGRADED) 2024-04-25T19:58:26.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:25 smithi132 bash[18607]: cluster 2024-04-25T19:58:25.143379+0000 mon.a (mon.0) 317 : cluster [WRN] Health check update: Degraded data redundancy: 22/66 objects degraded (33.333%), 13 pgs degraded (PG_DEGRADED) 2024-04-25T19:58:27.841 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:58:28.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:27 smithi132 bash[18607]: cluster 2024-04-25T19:58:26.750519+0000 mgr.x (mgr.14152) 168 : cluster [DBG] pgmap v151: 64 pgs: 13 active+undersized+degraded, 51 active+undersized; 2.4 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 6.1 KiB/s rd, 1.6 KiB/s wr, 8 op/s; 22/66 objects degraded (33.333%) 2024-04-25T19:58:28.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:27 smithi132 bash[18607]: cluster 2024-04-25T19:58:26.750519+0000 mgr.x (mgr.14152) 168 : cluster [DBG] pgmap v151: 64 pgs: 13 active+undersized+degraded, 51 active+undersized; 2.4 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 6.1 KiB/s rd, 1.6 KiB/s wr, 8 op/s; 22/66 objects degraded (33.333%) 2024-04-25T19:58:30.522 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:30 smithi132 bash[18607]: cluster 2024-04-25T19:58:28.751326+0000 mgr.x (mgr.14152) 169 : cluster [DBG] pgmap v152: 64 pgs: 13 active+undersized+degraded, 51 active+undersized; 2.4 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 8.2 KiB/s rd, 1.3 KiB/s wr, 10 op/s; 22/66 objects degraded (33.333%) 2024-04-25T19:58:30.522 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:30 smithi132 bash[18607]: cluster 2024-04-25T19:58:28.751326+0000 mgr.x (mgr.14152) 169 : cluster [DBG] pgmap v152: 64 pgs: 13 active+undersized+degraded, 51 active+undersized; 2.4 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 8.2 KiB/s rd, 1.3 KiB/s wr, 10 op/s; 22/66 objects degraded (33.333%) 2024-04-25T19:58:30.522 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:30 smithi132 bash[18607]: audit 2024-04-25T19:58:29.204074+0000 mon.a (mon.0) 318 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:58:30.522 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:30 smithi132 bash[18607]: audit 2024-04-25T19:58:29.204074+0000 mon.a (mon.0) 318 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:58:30.522 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:30 smithi132 bash[18607]: audit 2024-04-25T19:58:29.207644+0000 mon.a (mon.0) 319 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:58:30.522 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:30 smithi132 bash[18607]: audit 2024-04-25T19:58:29.207644+0000 mon.a (mon.0) 319 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:58:30.522 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:30 smithi132 bash[18607]: audit 2024-04-25T19:58:29.208698+0000 mon.a (mon.0) 320 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:58:30.522 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:30 smithi132 bash[18607]: audit 2024-04-25T19:58:29.208698+0000 mon.a (mon.0) 320 : audit [DBG] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:58:30.522 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:30 smithi132 bash[18607]: audit 2024-04-25T19:58:29.209564+0000 mon.a (mon.0) 321 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:58:30.523 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:30 smithi132 bash[18607]: audit 2024-04-25T19:58:29.209564+0000 mon.a (mon.0) 321 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:58:30.523 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:30 smithi132 bash[18607]: audit 2024-04-25T19:58:29.213227+0000 mon.a (mon.0) 322 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:58:30.523 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:30 smithi132 bash[18607]: audit 2024-04-25T19:58:29.213227+0000 mon.a (mon.0) 322 : audit [INF] from='mgr.14152 172.21.15.132:0/3882764922' entity='mgr.x' 2024-04-25T19:58:30.548 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T19:58:30.548 INFO:teuthology.orchestra.run.smithi132.stdout:[{"events": ["2024-04-25T19:58:18.434581Z service:mds.cephfs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:58:29.200474Z", "running": 2, "size": 2}}, {"events": ["2024-04-25T19:54:53.573757Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:58:29.200126Z", "running": 1, "size": 1}}, {"events": ["2024-04-25T19:54:28.767400Z service:mon [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:58:29.199988Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:58:29.200244Z", "running": 2, "size": 2}, "unmanaged": true}] 2024-04-25T19:58:31.174 INFO:tasks.cephadm:mds.cephfs has 2/2 2024-04-25T19:58:31.174 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-25T19:58:31.184 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi132.front.sepia.ceph.com 2024-04-25T19:58:31.184 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- bash -c 'ceph fs subvolumegroup create cephfs smb' 2024-04-25T19:58:32.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:32 smithi132 bash[18607]: audit 2024-04-25T19:58:30.546638+0000 mgr.x (mgr.14152) 170 : audit [DBG] from='client.14242 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T19:58:32.522 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:32 smithi132 bash[18607]: audit 2024-04-25T19:58:30.546638+0000 mgr.x (mgr.14152) 170 : audit [DBG] from='client.14242 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T19:58:32.522 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:32 smithi132 bash[18607]: cluster 2024-04-25T19:58:30.752222+0000 mgr.x (mgr.14152) 171 : cluster [DBG] pgmap v153: 64 pgs: 13 active+undersized+degraded, 51 active+undersized; 2.4 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 8.2 KiB/s rd, 0 B/s wr, 8 op/s; 22/66 objects degraded (33.333%) 2024-04-25T19:58:32.522 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:32 smithi132 bash[18607]: cluster 2024-04-25T19:58:30.752222+0000 mgr.x (mgr.14152) 171 : cluster [DBG] pgmap v153: 64 pgs: 13 active+undersized+degraded, 51 active+undersized; 2.4 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 8.2 KiB/s rd, 0 B/s wr, 8 op/s; 22/66 objects degraded (33.333%) 2024-04-25T19:58:34.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:34 smithi132 bash[18607]: cluster 2024-04-25T19:58:32.753024+0000 mgr.x (mgr.14152) 172 : cluster [DBG] pgmap v154: 64 pgs: 13 active+undersized+degraded, 51 active+undersized; 2.4 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 7.0 KiB/s rd, 0 B/s wr, 6 op/s; 22/66 objects degraded (33.333%) 2024-04-25T19:58:34.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:34 smithi132 bash[18607]: cluster 2024-04-25T19:58:32.753024+0000 mgr.x (mgr.14152) 172 : cluster [DBG] pgmap v154: 64 pgs: 13 active+undersized+degraded, 51 active+undersized; 2.4 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 7.0 KiB/s rd, 0 B/s wr, 6 op/s; 22/66 objects degraded (33.333%) 2024-04-25T19:58:35.924 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:58:36.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:36 smithi132 bash[18607]: cluster 2024-04-25T19:58:34.754040+0000 mgr.x (mgr.14152) 173 : cluster [DBG] pgmap v155: 64 pgs: 13 active+undersized+degraded, 51 active+undersized; 2.4 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 6.8 KiB/s rd, 0 B/s wr, 6 op/s; 22/66 objects degraded (33.333%) 2024-04-25T19:58:36.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:36 smithi132 bash[18607]: cluster 2024-04-25T19:58:34.754040+0000 mgr.x (mgr.14152) 173 : cluster [DBG] pgmap v155: 64 pgs: 13 active+undersized+degraded, 51 active+undersized; 2.4 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 6.8 KiB/s rd, 0 B/s wr, 6 op/s; 22/66 objects degraded (33.333%) 2024-04-25T19:58:38.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:38 smithi132 bash[18607]: cluster 2024-04-25T19:58:36.754608+0000 mgr.x (mgr.14152) 174 : cluster [DBG] pgmap v156: 64 pgs: 13 active+undersized+degraded, 51 active+undersized; 2.4 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 2.7 KiB/s rd, 0 B/s wr, 2 op/s; 22/66 objects degraded (33.333%) 2024-04-25T19:58:38.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:38 smithi132 bash[18607]: cluster 2024-04-25T19:58:36.754608+0000 mgr.x (mgr.14152) 174 : cluster [DBG] pgmap v156: 64 pgs: 13 active+undersized+degraded, 51 active+undersized; 2.4 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 2.7 KiB/s rd, 0 B/s wr, 2 op/s; 22/66 objects degraded (33.333%) 2024-04-25T19:58:39.021 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:58:38 smithi132 bash[18860]: debug 2024-04-25T19:58:38.524+0000 7f7692126700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:58:39.022 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:58:38 smithi132 bash[18860]: debug 2024-04-25T19:58:38.524+0000 7f7692126700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:58:39.022 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:58:38 smithi132 bash[18860]: debug 2024-04-25T19:58:38.524+0000 7f7692126700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:58:39.022 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:58:38 smithi132 bash[18860]: debug 2024-04-25T19:58:38.524+0000 7f7692126700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:58:39.022 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:58:38 smithi132 bash[18860]: debug 2024-04-25T19:58:38.524+0000 7f7692126700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:58:39.156 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- bash -c 'ceph fs subvolume create cephfs sv1 --group-name=smb --mode=0777' 2024-04-25T19:58:40.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:40 smithi132 bash[18607]: audit 2024-04-25T19:58:38.520207+0000 mgr.x (mgr.14152) 175 : audit [DBG] from='client.14244 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "smb", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:58:40.522 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:40 smithi132 bash[18607]: audit 2024-04-25T19:58:38.520207+0000 mgr.x (mgr.14152) 175 : audit [DBG] from='client.14244 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "smb", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:58:40.522 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:40 smithi132 bash[18607]: cluster 2024-04-25T19:58:38.755426+0000 mgr.x (mgr.14152) 176 : cluster [DBG] pgmap v157: 64 pgs: 13 active+undersized+degraded, 51 active+undersized; 2.4 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 2.7 KiB/s rd, 0 B/s wr, 2 op/s; 22/66 objects degraded (33.333%) 2024-04-25T19:58:40.522 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:40 smithi132 bash[18607]: cluster 2024-04-25T19:58:38.755426+0000 mgr.x (mgr.14152) 176 : cluster [DBG] pgmap v157: 64 pgs: 13 active+undersized+degraded, 51 active+undersized; 2.4 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 2.7 KiB/s rd, 0 B/s wr, 2 op/s; 22/66 objects degraded (33.333%) 2024-04-25T19:58:40.522 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:40 smithi132 bash[18607]: cluster 2024-04-25T19:58:39.223910+0000 mon.a (mon.0) 323 : cluster [DBG] mgrmap e14: x(active, since 4m) 2024-04-25T19:58:40.522 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:40 smithi132 bash[18607]: cluster 2024-04-25T19:58:39.223910+0000 mon.a (mon.0) 323 : cluster [DBG] mgrmap e14: x(active, since 4m) 2024-04-25T19:58:42.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:42 smithi132 bash[18607]: cluster 2024-04-25T19:58:40.756392+0000 mgr.x (mgr.14152) 177 : cluster [DBG] pgmap v158: 64 pgs: 13 active+undersized+degraded, 51 active+undersized; 8.8 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 852 B/s wr, 0 op/s; 22/66 objects degraded (33.333%) 2024-04-25T19:58:42.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:42 smithi132 bash[18607]: cluster 2024-04-25T19:58:40.756392+0000 mgr.x (mgr.14152) 177 : cluster [DBG] pgmap v158: 64 pgs: 13 active+undersized+degraded, 51 active+undersized; 8.8 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 852 B/s wr, 0 op/s; 22/66 objects degraded (33.333%) 2024-04-25T19:58:43.902 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:58:44.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:44 smithi132 bash[18607]: cluster 2024-04-25T19:58:42.757153+0000 mgr.x (mgr.14152) 178 : cluster [DBG] pgmap v159: 64 pgs: 13 active+undersized+degraded, 51 active+undersized; 8.8 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 852 B/s wr, 0 op/s; 22/66 objects degraded (33.333%) 2024-04-25T19:58:44.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:44 smithi132 bash[18607]: cluster 2024-04-25T19:58:42.757153+0000 mgr.x (mgr.14152) 178 : cluster [DBG] pgmap v159: 64 pgs: 13 active+undersized+degraded, 51 active+undersized; 8.8 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 852 B/s wr, 0 op/s; 22/66 objects degraded (33.333%) 2024-04-25T19:58:46.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:46 smithi132 bash[18607]: cluster 2024-04-25T19:58:44.757707+0000 mgr.x (mgr.14152) 179 : cluster [DBG] pgmap v160: 64 pgs: 13 active+undersized+degraded, 51 active+undersized; 8.8 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 852 B/s wr, 0 op/s; 22/66 objects degraded (33.333%) 2024-04-25T19:58:46.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:46 smithi132 bash[18607]: cluster 2024-04-25T19:58:44.757707+0000 mgr.x (mgr.14152) 179 : cluster [DBG] pgmap v160: 64 pgs: 13 active+undersized+degraded, 51 active+undersized; 8.8 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 852 B/s wr, 0 op/s; 22/66 objects degraded (33.333%) 2024-04-25T19:58:47.205 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- bash -c 'ceph fs subvolume create cephfs sv2 --group-name=smb --mode=0777' 2024-04-25T19:58:48.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:48 smithi132 bash[18607]: audit 2024-04-25T19:58:46.546904+0000 mgr.x (mgr.14152) 180 : audit [DBG] from='client.14248 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "group_name": "smb", "mode": "0777", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:58:48.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:48 smithi132 bash[18607]: audit 2024-04-25T19:58:46.546904+0000 mgr.x (mgr.14152) 180 : audit [DBG] from='client.14248 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "group_name": "smb", "mode": "0777", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:58:48.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:48 smithi132 bash[18607]: cluster 2024-04-25T19:58:46.758421+0000 mgr.x (mgr.14152) 181 : cluster [DBG] pgmap v161: 64 pgs: 13 active+undersized+degraded, 51 active+undersized; 8.8 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 852 B/s wr, 0 op/s; 22/66 objects degraded (33.333%) 2024-04-25T19:58:48.522 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:48 smithi132 bash[18607]: cluster 2024-04-25T19:58:46.758421+0000 mgr.x (mgr.14152) 181 : cluster [DBG] pgmap v161: 64 pgs: 13 active+undersized+degraded, 51 active+undersized; 8.8 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 852 B/s wr, 0 op/s; 22/66 objects degraded (33.333%) 2024-04-25T19:58:50.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:50 smithi132 bash[18607]: cluster 2024-04-25T19:58:48.759299+0000 mgr.x (mgr.14152) 182 : cluster [DBG] pgmap v162: 64 pgs: 13 active+undersized+degraded, 51 active+undersized; 8.8 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 853 B/s wr, 0 op/s; 22/66 objects degraded (33.333%) 2024-04-25T19:58:50.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:50 smithi132 bash[18607]: cluster 2024-04-25T19:58:48.759299+0000 mgr.x (mgr.14152) 182 : cluster [DBG] pgmap v162: 64 pgs: 13 active+undersized+degraded, 51 active+undersized; 8.8 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 853 B/s wr, 0 op/s; 22/66 objects degraded (33.333%) 2024-04-25T19:58:51.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:51 smithi132 bash[18607]: cluster 2024-04-25T19:58:51.226882+0000 mon.a (mon.0) 324 : cluster [WRN] Health check update: Degraded data redundancy: 23/69 objects degraded (33.333%), 13 pgs degraded (PG_DEGRADED) 2024-04-25T19:58:51.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:51 smithi132 bash[18607]: cluster 2024-04-25T19:58:51.226882+0000 mon.a (mon.0) 324 : cluster [WRN] Health check update: Degraded data redundancy: 23/69 objects degraded (33.333%), 13 pgs degraded (PG_DEGRADED) 2024-04-25T19:58:51.949 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:58:52.690 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:52 smithi132 bash[18607]: cluster 2024-04-25T19:58:50.760263+0000 mgr.x (mgr.14152) 183 : cluster [DBG] pgmap v163: 64 pgs: 13 active+undersized+degraded, 51 active+undersized; 24 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 2.3 KiB/s wr, 1 op/s; 23/69 objects degraded (33.333%) 2024-04-25T19:58:52.690 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:52 smithi132 bash[18607]: cluster 2024-04-25T19:58:50.760263+0000 mgr.x (mgr.14152) 183 : cluster [DBG] pgmap v163: 64 pgs: 13 active+undersized+degraded, 51 active+undersized; 24 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 2.3 KiB/s wr, 1 op/s; 23/69 objects degraded (33.333%) 2024-04-25T19:58:54.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:54 smithi132 bash[18607]: cluster 2024-04-25T19:58:52.760731+0000 mgr.x (mgr.14152) 184 : cluster [DBG] pgmap v164: 64 pgs: 13 active+undersized+degraded, 51 active+undersized; 24 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 1.5 KiB/s wr, 0 op/s; 23/69 objects degraded (33.333%) 2024-04-25T19:58:54.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:54 smithi132 bash[18607]: cluster 2024-04-25T19:58:52.760731+0000 mgr.x (mgr.14152) 184 : cluster [DBG] pgmap v164: 64 pgs: 13 active+undersized+degraded, 51 active+undersized; 24 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 1.5 KiB/s wr, 0 op/s; 23/69 objects degraded (33.333%) 2024-04-25T19:58:55.179 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- bash -c 'ceph mgr module enable smb' 2024-04-25T19:58:56.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:56 smithi132 bash[18607]: audit 2024-04-25T19:58:54.503610+0000 mgr.x (mgr.14152) 185 : audit [DBG] from='client.14250 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv2", "group_name": "smb", "mode": "0777", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:58:56.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:56 smithi132 bash[18607]: audit 2024-04-25T19:58:54.503610+0000 mgr.x (mgr.14152) 185 : audit [DBG] from='client.14250 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv2", "group_name": "smb", "mode": "0777", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:58:56.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:56 smithi132 bash[18607]: cluster 2024-04-25T19:58:54.761513+0000 mgr.x (mgr.14152) 186 : cluster [DBG] pgmap v165: 64 pgs: 13 active+undersized+degraded, 51 active+undersized; 28 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 2.0 KiB/s wr, 1 op/s; 23/69 objects degraded (33.333%) 2024-04-25T19:58:56.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:56 smithi132 bash[18607]: cluster 2024-04-25T19:58:54.761513+0000 mgr.x (mgr.14152) 186 : cluster [DBG] pgmap v165: 64 pgs: 13 active+undersized+degraded, 51 active+undersized; 28 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 2.0 KiB/s wr, 1 op/s; 23/69 objects degraded (33.333%) 2024-04-25T19:58:58.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:58 smithi132 bash[18607]: cluster 2024-04-25T19:58:56.762232+0000 mgr.x (mgr.14152) 187 : cluster [DBG] pgmap v166: 64 pgs: 13 active+undersized+degraded, 51 active+undersized; 28 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 2.0 KiB/s wr, 1 op/s; 23/69 objects degraded (33.333%) 2024-04-25T19:58:58.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:58:58 smithi132 bash[18607]: cluster 2024-04-25T19:58:56.762232+0000 mgr.x (mgr.14152) 187 : cluster [DBG] pgmap v166: 64 pgs: 13 active+undersized+degraded, 51 active+undersized; 28 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 2.0 KiB/s wr, 1 op/s; 23/69 objects degraded (33.333%) 2024-04-25T19:58:59.928 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:59:00.705 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:00 smithi132 bash[18607]: cluster 2024-04-25T19:58:58.763124+0000 mgr.x (mgr.14152) 188 : cluster [DBG] pgmap v167: 64 pgs: 13 active+undersized+degraded, 51 active+undersized; 28 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 2.0 KiB/s wr, 1 op/s; 23/69 objects degraded (33.333%) 2024-04-25T19:59:00.705 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:00 smithi132 bash[18607]: cluster 2024-04-25T19:58:58.763124+0000 mgr.x (mgr.14152) 188 : cluster [DBG] pgmap v167: 64 pgs: 13 active+undersized+degraded, 51 active+undersized; 28 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 2.0 KiB/s wr, 1 op/s; 23/69 objects degraded (33.333%) 2024-04-25T19:59:01.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:01 smithi132 bash[18607]: cluster 2024-04-25T19:59:01.287524+0000 mon.a (mon.0) 325 : cluster [WRN] Health check update: Degraded data redundancy: 24/72 objects degraded (33.333%), 13 pgs degraded (PG_DEGRADED) 2024-04-25T19:59:01.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:01 smithi132 bash[18607]: cluster 2024-04-25T19:59:01.287524+0000 mon.a (mon.0) 325 : cluster [WRN] Health check update: Degraded data redundancy: 24/72 objects degraded (33.333%), 13 pgs degraded (PG_DEGRADED) 2024-04-25T19:59:02.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:02 smithi132 bash[18607]: cluster 2024-04-25T19:59:00.763705+0000 mgr.x (mgr.14152) 189 : cluster [DBG] pgmap v168: 64 pgs: 13 active+undersized+degraded, 51 active+undersized; 55 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 4.7 KiB/s wr, 2 op/s; 24/72 objects degraded (33.333%) 2024-04-25T19:59:02.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:02 smithi132 bash[18607]: cluster 2024-04-25T19:59:00.763705+0000 mgr.x (mgr.14152) 189 : cluster [DBG] pgmap v168: 64 pgs: 13 active+undersized+degraded, 51 active+undersized; 55 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 4.7 KiB/s wr, 2 op/s; 24/72 objects degraded (33.333%) 2024-04-25T19:59:03.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:03 smithi132 bash[18607]: audit 2024-04-25T19:59:02.667132+0000 mon.a (mon.0) 326 : audit [INF] from='client.? 172.21.15.132:0/3358253726' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "smb"}]: dispatch 2024-04-25T19:59:03.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:03 smithi132 bash[18607]: audit 2024-04-25T19:59:02.667132+0000 mon.a (mon.0) 326 : audit [INF] from='client.? 172.21.15.132:0/3358253726' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "smb"}]: dispatch 2024-04-25T19:59:03.771 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:03 smithi132 bash[18860]: ignoring --setuser ceph since I am not root 2024-04-25T19:59:03.771 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:03 smithi132 bash[18860]: ignoring --setgroup ceph since I am not root 2024-04-25T19:59:03.771 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:03 smithi132 bash[18860]: debug 2024-04-25T19:59:03.551+0000 7f918e227200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T19:59:04.060 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- bash -c 'sleep 30' 2024-04-25T19:59:04.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:04 smithi132 bash[18607]: audit 2024-04-25T19:59:03.348966+0000 mon.a (mon.0) 327 : audit [INF] from='client.? 172.21.15.132:0/3358253726' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "smb"}]': finished 2024-04-25T19:59:04.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:04 smithi132 bash[18607]: audit 2024-04-25T19:59:03.348966+0000 mon.a (mon.0) 327 : audit [INF] from='client.? 172.21.15.132:0/3358253726' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "smb"}]': finished 2024-04-25T19:59:04.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:04 smithi132 bash[18607]: cluster 2024-04-25T19:59:03.355598+0000 mon.a (mon.0) 328 : cluster [DBG] mgrmap e15: x(active, since 5m) 2024-04-25T19:59:04.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:04 smithi132 bash[18607]: cluster 2024-04-25T19:59:03.355598+0000 mon.a (mon.0) 328 : cluster [DBG] mgrmap e15: x(active, since 5m) 2024-04-25T19:59:05.471 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:05 smithi132 bash[18860]: debug 2024-04-25T19:59:05.123+0000 7f918e227200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T19:59:05.471 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:05 smithi132 bash[18860]: debug 2024-04-25T19:59:05.223+0000 7f918e227200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T19:59:06.521 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:05 smithi132 bash[18860]: debug 2024-04-25T19:59:05.471+0000 7f918e227200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T19:59:06.521 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:06 smithi132 bash[18860]: debug 2024-04-25T19:59:06.267+0000 7f918e227200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T19:59:06.521 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:06 smithi132 bash[18860]: debug 2024-04-25T19:59:06.379+0000 7f918e227200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T19:59:06.521 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:06 smithi132 bash[18860]: debug 2024-04-25T19:59:06.483+0000 7f918e227200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T19:59:07.271 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:06 smithi132 bash[18860]: debug 2024-04-25T19:59:06.843+0000 7f918e227200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T19:59:07.271 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:06 smithi132 bash[18860]: debug 2024-04-25T19:59:06.955+0000 7f918e227200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T19:59:07.271 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:07 smithi132 bash[18860]: debug 2024-04-25T19:59:07.059+0000 7f918e227200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T19:59:07.770 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:07 smithi132 bash[18860]: debug 2024-04-25T19:59:07.419+0000 7f918e227200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T19:59:07.771 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:07 smithi132 bash[18860]: debug 2024-04-25T19:59:07.663+0000 7f918e227200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T19:59:08.270 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:07 smithi132 bash[18860]: debug 2024-04-25T19:59:07.923+0000 7f918e227200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T19:59:08.271 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:08 smithi132 bash[18860]: debug 2024-04-25T19:59:08.039+0000 7f918e227200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T19:59:08.731 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:08 smithi132 bash[18860]: debug 2024-04-25T19:59:08.315+0000 7f918e227200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T19:59:08.778 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:59:09.021 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:08 smithi132 bash[18860]: debug 2024-04-25T19:59:08.731+0000 7f918e227200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T19:59:09.021 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:08 smithi132 bash[18860]: debug 2024-04-25T19:59:08.975+0000 7f918e227200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T19:59:09.346 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:09 smithi132 bash[18860]: debug 2024-04-25T19:59:09.095+0000 7f918e227200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T19:59:09.618 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:09 smithi132 bash[18860]: debug 2024-04-25T19:59:09.347+0000 7f918e227200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T19:59:10.770 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:10 smithi132 bash[18860]: debug 2024-04-25T19:59:10.391+0000 7f918e227200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T19:59:10.771 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:10 smithi132 bash[18860]: debug 2024-04-25T19:59:10.507+0000 7f918e227200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T19:59:11.215 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:10 smithi132 bash[18860]: debug 2024-04-25T19:59:10.811+0000 7f918e227200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T19:59:11.215 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:10 smithi132 bash[18860]: debug 2024-04-25T19:59:10.927+0000 7f918e227200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T19:59:11.520 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:11 smithi132 bash[18860]: debug 2024-04-25T19:59:11.215+0000 7f918e227200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-04-25T19:59:11.982 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:11 smithi132 bash[18860]: debug 2024-04-25T19:59:11.679+0000 7f918e227200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T19:59:11.982 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:11 smithi132 bash[18860]: debug 2024-04-25T19:59:11.803+0000 7f918e227200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T19:59:11.983 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:11 smithi132 bash[18607]: cluster 2024-04-25T19:59:11.826344+0000 mon.a (mon.0) 329 : cluster [INF] Active manager daemon x restarted 2024-04-25T19:59:11.983 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:11 smithi132 bash[18607]: cluster 2024-04-25T19:59:11.826344+0000 mon.a (mon.0) 329 : cluster [INF] Active manager daemon x restarted 2024-04-25T19:59:11.983 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:11 smithi132 bash[18607]: cluster 2024-04-25T19:59:11.827035+0000 mon.a (mon.0) 330 : cluster [INF] Activating manager daemon x 2024-04-25T19:59:11.983 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:11 smithi132 bash[18607]: cluster 2024-04-25T19:59:11.827035+0000 mon.a (mon.0) 330 : cluster [INF] Activating manager daemon x 2024-04-25T19:59:11.983 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:11 smithi132 bash[18607]: cluster 2024-04-25T19:59:11.833503+0000 mon.a (mon.0) 331 : cluster [DBG] osdmap e21: 2 total, 2 up, 2 in 2024-04-25T19:59:11.983 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:11 smithi132 bash[18607]: cluster 2024-04-25T19:59:11.833503+0000 mon.a (mon.0) 331 : cluster [DBG] osdmap e21: 2 total, 2 up, 2 in 2024-04-25T19:59:11.983 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:11 smithi132 bash[18607]: cluster 2024-04-25T19:59:11.834083+0000 mon.a (mon.0) 332 : cluster [DBG] mgrmap e16: x(active, starting, since 0.00725959s) 2024-04-25T19:59:11.983 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:11 smithi132 bash[18607]: cluster 2024-04-25T19:59:11.834083+0000 mon.a (mon.0) 332 : cluster [DBG] mgrmap e16: x(active, starting, since 0.00725959s) 2024-04-25T19:59:11.983 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:11 smithi132 bash[18607]: audit 2024-04-25T19:59:11.837221+0000 mon.a (mon.0) 333 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T19:59:11.984 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:11 smithi132 bash[18607]: audit 2024-04-25T19:59:11.837221+0000 mon.a (mon.0) 333 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T19:59:11.984 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:11 smithi132 bash[18607]: audit 2024-04-25T19:59:11.837666+0000 mon.a (mon.0) 334 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi132.wyahux"}]: dispatch 2024-04-25T19:59:11.984 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:11 smithi132 bash[18607]: audit 2024-04-25T19:59:11.837666+0000 mon.a (mon.0) 334 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi132.wyahux"}]: dispatch 2024-04-25T19:59:11.984 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:11 smithi132 bash[18607]: audit 2024-04-25T19:59:11.838052+0000 mon.a (mon.0) 335 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi132.wpuwhp"}]: dispatch 2024-04-25T19:59:11.984 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:11 smithi132 bash[18607]: audit 2024-04-25T19:59:11.838052+0000 mon.a (mon.0) 335 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi132.wpuwhp"}]: dispatch 2024-04-25T19:59:11.984 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:11 smithi132 bash[18607]: audit 2024-04-25T19:59:11.838471+0000 mon.a (mon.0) 336 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T19:59:11.984 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:11 smithi132 bash[18607]: audit 2024-04-25T19:59:11.838471+0000 mon.a (mon.0) 336 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T19:59:11.984 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:11 smithi132 bash[18607]: audit 2024-04-25T19:59:11.838668+0000 mon.a (mon.0) 337 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T19:59:11.984 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:11 smithi132 bash[18607]: audit 2024-04-25T19:59:11.838668+0000 mon.a (mon.0) 337 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T19:59:11.984 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:11 smithi132 bash[18607]: audit 2024-04-25T19:59:11.838963+0000 mon.a (mon.0) 338 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T19:59:11.984 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:11 smithi132 bash[18607]: audit 2024-04-25T19:59:11.838963+0000 mon.a (mon.0) 338 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T19:59:11.984 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:11 smithi132 bash[18607]: audit 2024-04-25T19:59:11.839279+0000 mon.a (mon.0) 339 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T19:59:11.984 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:11 smithi132 bash[18607]: audit 2024-04-25T19:59:11.839279+0000 mon.a (mon.0) 339 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T19:59:11.984 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:11 smithi132 bash[18607]: audit 2024-04-25T19:59:11.839649+0000 mon.a (mon.0) 340 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T19:59:11.985 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:11 smithi132 bash[18607]: audit 2024-04-25T19:59:11.839649+0000 mon.a (mon.0) 340 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T19:59:11.985 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:11 smithi132 bash[18607]: audit 2024-04-25T19:59:11.840058+0000 mon.a (mon.0) 341 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T19:59:11.985 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:11 smithi132 bash[18607]: audit 2024-04-25T19:59:11.840058+0000 mon.a (mon.0) 341 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T19:59:11.985 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:11 smithi132 bash[18607]: cluster 2024-04-25T19:59:11.868529+0000 mon.a (mon.0) 342 : cluster [INF] Manager daemon x is now available 2024-04-25T19:59:11.985 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:11 smithi132 bash[18607]: cluster 2024-04-25T19:59:11.868529+0000 mon.a (mon.0) 342 : cluster [INF] Manager daemon x is now available 2024-04-25T19:59:13.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:12 smithi132 bash[18607]: audit 2024-04-25T19:59:11.915669+0000 mon.a (mon.0) 343 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:59:13.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:12 smithi132 bash[18607]: audit 2024-04-25T19:59:11.915669+0000 mon.a (mon.0) 343 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:59:13.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:12 smithi132 bash[18607]: audit 2024-04-25T19:59:11.966675+0000 mon.a (mon.0) 344 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "osd pool create", "pool": ".smb", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T19:59:13.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:12 smithi132 bash[18607]: audit 2024-04-25T19:59:11.966675+0000 mon.a (mon.0) 344 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "osd pool create", "pool": ".smb", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T19:59:13.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:12 smithi132 bash[18607]: audit 2024-04-25T19:59:11.979440+0000 mon.a (mon.0) 345 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T19:59:13.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:12 smithi132 bash[18607]: audit 2024-04-25T19:59:11.979440+0000 mon.a (mon.0) 345 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T19:59:13.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:12 smithi132 bash[18607]: audit 2024-04-25T19:59:11.990106+0000 mon.a (mon.0) 346 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T19:59:13.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:12 smithi132 bash[18607]: audit 2024-04-25T19:59:11.990106+0000 mon.a (mon.0) 346 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T19:59:13.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:12 smithi132 bash[18607]: cluster 2024-04-25T19:59:12.844431+0000 mon.a (mon.0) 347 : cluster [DBG] mgrmap e17: x(active, since 1.01759s) 2024-04-25T19:59:13.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:12 smithi132 bash[18607]: cluster 2024-04-25T19:59:12.844431+0000 mon.a (mon.0) 347 : cluster [DBG] mgrmap e17: x(active, since 1.01759s) 2024-04-25T19:59:14.271 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:13 smithi132 bash[18860]: debug 2024-04-25T19:59:13.951+0000 7f90dbf33700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:59:14.271 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:13 smithi132 bash[18860]: debug 2024-04-25T19:59:13.951+0000 7f90dbf33700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:59:14.271 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:13 smithi132 bash[18860]: debug 2024-04-25T19:59:13.951+0000 7f90dbf33700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:59:14.271 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:13 smithi132 bash[18860]: debug 2024-04-25T19:59:13.951+0000 7f90dbf33700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:59:14.271 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:13 smithi132 bash[18860]: debug 2024-04-25T19:59:13.951+0000 7f90dbf33700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:59:14.271 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:13 smithi132 bash[18860]: debug 2024-04-25T19:59:13.951+0000 7f90de738700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:59:14.272 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:13 smithi132 bash[18860]: debug 2024-04-25T19:59:13.951+0000 7f90de738700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:59:14.272 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:13 smithi132 bash[18860]: debug 2024-04-25T19:59:13.951+0000 7f90de738700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:59:14.272 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:13 smithi132 bash[18860]: debug 2024-04-25T19:59:13.951+0000 7f90de738700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:59:14.272 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:13 smithi132 bash[18860]: debug 2024-04-25T19:59:13.951+0000 7f90de738700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:59:14.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:13 smithi132 bash[18607]: cephadm 2024-04-25T19:59:12.614856+0000 mgr.x (mgr.14254) 1 : cephadm [INF] [25/Apr/2024:19:59:12] ENGINE Bus STARTING 2024-04-25T19:59:14.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:13 smithi132 bash[18607]: cephadm 2024-04-25T19:59:12.614856+0000 mgr.x (mgr.14254) 1 : cephadm [INF] [25/Apr/2024:19:59:12] ENGINE Bus STARTING 2024-04-25T19:59:14.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:13 smithi132 bash[18607]: cephadm 2024-04-25T19:59:12.716854+0000 mgr.x (mgr.14254) 2 : cephadm [INF] [25/Apr/2024:19:59:12] ENGINE Serving on http://172.21.15.132:8765 2024-04-25T19:59:14.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:13 smithi132 bash[18607]: cephadm 2024-04-25T19:59:12.716854+0000 mgr.x (mgr.14254) 2 : cephadm [INF] [25/Apr/2024:19:59:12] ENGINE Serving on http://172.21.15.132:8765 2024-04-25T19:59:14.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:13 smithi132 bash[18607]: cephadm 2024-04-25T19:59:12.838592+0000 mgr.x (mgr.14254) 3 : cephadm [INF] [25/Apr/2024:19:59:12] ENGINE Serving on https://172.21.15.132:7150 2024-04-25T19:59:14.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:13 smithi132 bash[18607]: cephadm 2024-04-25T19:59:12.838592+0000 mgr.x (mgr.14254) 3 : cephadm [INF] [25/Apr/2024:19:59:12] ENGINE Serving on https://172.21.15.132:7150 2024-04-25T19:59:14.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:13 smithi132 bash[18607]: cephadm 2024-04-25T19:59:12.838928+0000 mgr.x (mgr.14254) 4 : cephadm [INF] [25/Apr/2024:19:59:12] ENGINE Bus STARTED 2024-04-25T19:59:14.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:13 smithi132 bash[18607]: cephadm 2024-04-25T19:59:12.838928+0000 mgr.x (mgr.14254) 4 : cephadm [INF] [25/Apr/2024:19:59:12] ENGINE Bus STARTED 2024-04-25T19:59:14.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:13 smithi132 bash[18607]: cluster 2024-04-25T19:59:12.845618+0000 mgr.x (mgr.14254) 5 : cluster [DBG] pgmap v3: 64 pgs: 15 active+undersized+degraded, 49 active+undersized; 62 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 24/72 objects degraded (33.333%) 2024-04-25T19:59:14.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:13 smithi132 bash[18607]: cluster 2024-04-25T19:59:12.845618+0000 mgr.x (mgr.14254) 5 : cluster [DBG] pgmap v3: 64 pgs: 15 active+undersized+degraded, 49 active+undersized; 62 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 24/72 objects degraded (33.333%) 2024-04-25T19:59:14.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:13 smithi132 bash[18607]: audit 2024-04-25T19:59:12.888104+0000 mon.a (mon.0) 348 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd='[{"prefix": "osd pool create", "pool": ".smb", "yes_i_really_mean_it": true}]': finished 2024-04-25T19:59:14.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:13 smithi132 bash[18607]: audit 2024-04-25T19:59:12.888104+0000 mon.a (mon.0) 348 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd='[{"prefix": "osd pool create", "pool": ".smb", "yes_i_really_mean_it": true}]': finished 2024-04-25T19:59:14.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:13 smithi132 bash[18607]: cluster 2024-04-25T19:59:12.893182+0000 mon.a (mon.0) 349 : cluster [DBG] osdmap e22: 2 total, 2 up, 2 in 2024-04-25T19:59:14.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:13 smithi132 bash[18607]: cluster 2024-04-25T19:59:12.893182+0000 mon.a (mon.0) 349 : cluster [DBG] osdmap e22: 2 total, 2 up, 2 in 2024-04-25T19:59:14.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:13 smithi132 bash[18607]: audit 2024-04-25T19:59:12.896716+0000 mon.a (mon.0) 350 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "pool": ".smb", "app": "smb"}]: dispatch 2024-04-25T19:59:14.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:13 smithi132 bash[18607]: audit 2024-04-25T19:59:12.896716+0000 mon.a (mon.0) 350 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "pool": ".smb", "app": "smb"}]: dispatch 2024-04-25T19:59:14.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:13 smithi132 bash[18607]: cluster 2024-04-25T19:59:13.833366+0000 mon.a (mon.0) 351 : cluster [WRN] Health check update: Degraded data redundancy: 24/72 objects degraded (33.333%), 15 pgs degraded (PG_DEGRADED) 2024-04-25T19:59:14.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:13 smithi132 bash[18607]: cluster 2024-04-25T19:59:13.833366+0000 mon.a (mon.0) 351 : cluster [WRN] Health check update: Degraded data redundancy: 24/72 objects degraded (33.333%), 15 pgs degraded (PG_DEGRADED) 2024-04-25T19:59:15.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:14 smithi132 bash[18607]: cluster 2024-04-25T19:59:13.840174+0000 mgr.x (mgr.14254) 6 : cluster [DBG] pgmap v5: 96 pgs: 32 unknown, 15 active+undersized+degraded, 49 active+undersized; 62 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 24/72 objects degraded (33.333%) 2024-04-25T19:59:15.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:14 smithi132 bash[18607]: cluster 2024-04-25T19:59:13.840174+0000 mgr.x (mgr.14254) 6 : cluster [DBG] pgmap v5: 96 pgs: 32 unknown, 15 active+undersized+degraded, 49 active+undersized; 62 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 24/72 objects degraded (33.333%) 2024-04-25T19:59:15.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:14 smithi132 bash[18607]: audit 2024-04-25T19:59:13.892682+0000 mon.a (mon.0) 352 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "pool": ".smb", "app": "smb"}]': finished 2024-04-25T19:59:15.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:14 smithi132 bash[18607]: audit 2024-04-25T19:59:13.892682+0000 mon.a (mon.0) 352 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "pool": ".smb", "app": "smb"}]': finished 2024-04-25T19:59:15.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:14 smithi132 bash[18607]: cluster 2024-04-25T19:59:13.900035+0000 mon.a (mon.0) 353 : cluster [DBG] osdmap e23: 2 total, 2 up, 2 in 2024-04-25T19:59:15.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:14 smithi132 bash[18607]: cluster 2024-04-25T19:59:13.900035+0000 mon.a (mon.0) 353 : cluster [DBG] osdmap e23: 2 total, 2 up, 2 in 2024-04-25T19:59:15.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:14 smithi132 bash[18607]: cluster 2024-04-25T19:59:14.834484+0000 mon.a (mon.0) 354 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T19:59:15.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:14 smithi132 bash[18607]: cluster 2024-04-25T19:59:14.834484+0000 mon.a (mon.0) 354 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T19:59:15.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:14 smithi132 bash[18607]: cluster 2024-04-25T19:59:14.852019+0000 mon.a (mon.0) 355 : cluster [DBG] mgrmap e18: x(active, since 3s) 2024-04-25T19:59:15.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:14 smithi132 bash[18607]: cluster 2024-04-25T19:59:14.852019+0000 mon.a (mon.0) 355 : cluster [DBG] mgrmap e18: x(active, since 3s) 2024-04-25T19:59:16.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:15 smithi132 bash[18607]: cluster 2024-04-25T19:59:14.917756+0000 mon.a (mon.0) 356 : cluster [DBG] osdmap e24: 2 total, 2 up, 2 in 2024-04-25T19:59:16.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:15 smithi132 bash[18607]: cluster 2024-04-25T19:59:14.917756+0000 mon.a (mon.0) 356 : cluster [DBG] osdmap e24: 2 total, 2 up, 2 in 2024-04-25T19:59:17.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:16 smithi132 bash[18607]: cluster 2024-04-25T19:59:15.841741+0000 mgr.x (mgr.14254) 7 : cluster [DBG] pgmap v8: 96 pgs: 32 unknown, 15 active+undersized+degraded, 49 active+undersized; 62 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 24/72 objects degraded (33.333%) 2024-04-25T19:59:17.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:16 smithi132 bash[18607]: cluster 2024-04-25T19:59:15.841741+0000 mgr.x (mgr.14254) 7 : cluster [DBG] pgmap v8: 96 pgs: 32 unknown, 15 active+undersized+degraded, 49 active+undersized; 62 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 24/72 objects degraded (33.333%) 2024-04-25T19:59:17.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:16 smithi132 bash[18607]: cluster 2024-04-25T19:59:15.919427+0000 mon.a (mon.0) 357 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T19:59:17.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:16 smithi132 bash[18607]: cluster 2024-04-25T19:59:15.919427+0000 mon.a (mon.0) 357 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T19:59:17.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:16 smithi132 bash[18607]: cluster 2024-04-25T19:59:15.935930+0000 mon.a (mon.0) 358 : cluster [DBG] mgrmap e19: x(active, since 4s) 2024-04-25T19:59:17.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:16 smithi132 bash[18607]: cluster 2024-04-25T19:59:15.935930+0000 mon.a (mon.0) 358 : cluster [DBG] mgrmap e19: x(active, since 4s) 2024-04-25T19:59:19.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:18 smithi132 bash[18607]: cluster 2024-04-25T19:59:17.842514+0000 mgr.x (mgr.14254) 8 : cluster [DBG] pgmap v9: 96 pgs: 32 unknown, 15 active+undersized+degraded, 49 active+undersized; 62 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 24/72 objects degraded (33.333%) 2024-04-25T19:59:19.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:18 smithi132 bash[18607]: cluster 2024-04-25T19:59:17.842514+0000 mgr.x (mgr.14254) 8 : cluster [DBG] pgmap v9: 96 pgs: 32 unknown, 15 active+undersized+degraded, 49 active+undersized; 62 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 24/72 objects degraded (33.333%) 2024-04-25T19:59:19.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:18 smithi132 bash[18607]: audit 2024-04-25T19:59:17.912500+0000 mon.a (mon.0) 359 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T19:59:19.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:18 smithi132 bash[18607]: audit 2024-04-25T19:59:17.912500+0000 mon.a (mon.0) 359 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T19:59:19.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:18 smithi132 bash[18607]: audit 2024-04-25T19:59:17.918452+0000 mon.a (mon.0) 360 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T19:59:19.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:18 smithi132 bash[18607]: audit 2024-04-25T19:59:17.918452+0000 mon.a (mon.0) 360 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T19:59:20.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:20 smithi132 bash[18607]: audit 2024-04-25T19:59:19.015583+0000 mon.a (mon.0) 361 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T19:59:20.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:20 smithi132 bash[18607]: audit 2024-04-25T19:59:19.015583+0000 mon.a (mon.0) 361 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T19:59:20.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:20 smithi132 bash[18607]: audit 2024-04-25T19:59:19.022992+0000 mon.a (mon.0) 362 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T19:59:20.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:20 smithi132 bash[18607]: audit 2024-04-25T19:59:19.022992+0000 mon.a (mon.0) 362 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T19:59:20.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:20 smithi132 bash[18607]: audit 2024-04-25T19:59:19.024741+0000 mon.a (mon.0) 363 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:59:20.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:20 smithi132 bash[18607]: audit 2024-04-25T19:59:19.024741+0000 mon.a (mon.0) 363 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:59:20.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:20 smithi132 bash[18607]: audit 2024-04-25T19:59:19.026011+0000 mon.a (mon.0) 364 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:59:20.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:20 smithi132 bash[18607]: audit 2024-04-25T19:59:19.026011+0000 mon.a (mon.0) 364 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:59:20.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:20 smithi132 bash[18607]: cephadm 2024-04-25T19:59:19.026967+0000 mgr.x (mgr.14254) 9 : cephadm [INF] Adjusting osd_memory_target on smithi132 to 4536M 2024-04-25T19:59:20.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:20 smithi132 bash[18607]: cephadm 2024-04-25T19:59:19.026967+0000 mgr.x (mgr.14254) 9 : cephadm [INF] Adjusting osd_memory_target on smithi132 to 4536M 2024-04-25T19:59:20.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:20 smithi132 bash[18607]: audit 2024-04-25T19:59:19.029631+0000 mon.a (mon.0) 365 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T19:59:20.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:20 smithi132 bash[18607]: audit 2024-04-25T19:59:19.029631+0000 mon.a (mon.0) 365 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T19:59:20.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:20 smithi132 bash[18607]: audit 2024-04-25T19:59:19.031853+0000 mon.a (mon.0) 366 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:59:20.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:20 smithi132 bash[18607]: audit 2024-04-25T19:59:19.031853+0000 mon.a (mon.0) 366 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:59:20.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:20 smithi132 bash[18607]: audit 2024-04-25T19:59:19.032762+0000 mon.a (mon.0) 367 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:59:20.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:20 smithi132 bash[18607]: audit 2024-04-25T19:59:19.032762+0000 mon.a (mon.0) 367 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:59:20.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:20 smithi132 bash[18607]: cephadm 2024-04-25T19:59:19.034304+0000 mgr.x (mgr.14254) 10 : cephadm [INF] Updating smithi132:/etc/ceph/ceph.conf 2024-04-25T19:59:20.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:20 smithi132 bash[18607]: cephadm 2024-04-25T19:59:19.034304+0000 mgr.x (mgr.14254) 10 : cephadm [INF] Updating smithi132:/etc/ceph/ceph.conf 2024-04-25T19:59:20.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:20 smithi132 bash[18607]: cephadm 2024-04-25T19:59:19.249663+0000 mgr.x (mgr.14254) 11 : cephadm [INF] Updating smithi132:/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T19:59:20.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:20 smithi132 bash[18607]: cephadm 2024-04-25T19:59:19.249663+0000 mgr.x (mgr.14254) 11 : cephadm [INF] Updating smithi132:/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T19:59:20.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:20 smithi132 bash[18607]: cephadm 2024-04-25T19:59:19.414157+0000 mgr.x (mgr.14254) 12 : cephadm [INF] Updating smithi132:/etc/ceph/ceph.client.admin.keyring 2024-04-25T19:59:20.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:20 smithi132 bash[18607]: cephadm 2024-04-25T19:59:19.414157+0000 mgr.x (mgr.14254) 12 : cephadm [INF] Updating smithi132:/etc/ceph/ceph.client.admin.keyring 2024-04-25T19:59:20.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:20 smithi132 bash[18607]: audit 2024-04-25T19:59:19.749379+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T19:59:20.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:20 smithi132 bash[18607]: audit 2024-04-25T19:59:19.749379+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T19:59:20.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:20 smithi132 bash[18607]: audit 2024-04-25T19:59:19.755710+0000 mon.a (mon.0) 369 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T19:59:20.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:20 smithi132 bash[18607]: audit 2024-04-25T19:59:19.755710+0000 mon.a (mon.0) 369 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T19:59:20.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:20 smithi132 bash[18607]: audit 2024-04-25T19:59:19.762013+0000 mon.a (mon.0) 370 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T19:59:20.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:20 smithi132 bash[18607]: audit 2024-04-25T19:59:19.762013+0000 mon.a (mon.0) 370 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T19:59:20.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:20 smithi132 bash[18607]: audit 2024-04-25T19:59:19.801782+0000 mon.a (mon.0) 371 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:59:20.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:20 smithi132 bash[18607]: audit 2024-04-25T19:59:19.801782+0000 mon.a (mon.0) 371 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:59:20.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:20 smithi132 bash[18607]: audit 2024-04-25T19:59:19.805014+0000 mon.a (mon.0) 372 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:59:20.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:20 smithi132 bash[18607]: audit 2024-04-25T19:59:19.805014+0000 mon.a (mon.0) 372 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:59:20.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:20 smithi132 bash[18607]: audit 2024-04-25T19:59:19.806359+0000 mon.a (mon.0) 373 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:59:20.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:20 smithi132 bash[18607]: audit 2024-04-25T19:59:19.806359+0000 mon.a (mon.0) 373 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:59:20.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:20 smithi132 bash[18607]: audit 2024-04-25T19:59:19.814434+0000 mon.a (mon.0) 374 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T19:59:20.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:20 smithi132 bash[18607]: audit 2024-04-25T19:59:19.814434+0000 mon.a (mon.0) 374 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T19:59:21.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:21 smithi132 bash[18607]: cephadm 2024-04-25T19:59:19.579472+0000 mgr.x (mgr.14254) 13 : cephadm [INF] Updating smithi132:/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T19:59:21.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:21 smithi132 bash[18607]: cephadm 2024-04-25T19:59:19.579472+0000 mgr.x (mgr.14254) 13 : cephadm [INF] Updating smithi132:/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T19:59:21.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:21 smithi132 bash[18607]: cluster 2024-04-25T19:59:19.844288+0000 mgr.x (mgr.14254) 14 : cluster [DBG] pgmap v10: 96 pgs: 15 active+undersized+degraded, 81 active+undersized; 63 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 441 B/s wr, 0 op/s; 24/72 objects degraded (33.333%) 2024-04-25T19:59:21.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:21 smithi132 bash[18607]: cluster 2024-04-25T19:59:19.844288+0000 mgr.x (mgr.14254) 14 : cluster [DBG] pgmap v10: 96 pgs: 15 active+undersized+degraded, 81 active+undersized; 63 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 441 B/s wr, 0 op/s; 24/72 objects degraded (33.333%) 2024-04-25T19:59:23.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:23 smithi132 bash[18607]: cluster 2024-04-25T19:59:21.845743+0000 mgr.x (mgr.14254) 15 : cluster [DBG] pgmap v11: 96 pgs: 15 active+undersized+degraded, 81 active+undersized; 63 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 383 B/s wr, 0 op/s; 24/72 objects degraded (33.333%) 2024-04-25T19:59:23.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:23 smithi132 bash[18607]: cluster 2024-04-25T19:59:21.845743+0000 mgr.x (mgr.14254) 15 : cluster [DBG] pgmap v11: 96 pgs: 15 active+undersized+degraded, 81 active+undersized; 63 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 383 B/s wr, 0 op/s; 24/72 objects degraded (33.333%) 2024-04-25T19:59:24.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:24 smithi132 bash[18607]: cluster 2024-04-25T19:59:23.339473+0000 mon.a (mon.0) 375 : cluster [WRN] Health check update: Degraded data redundancy: 24/72 objects degraded (33.333%), 15 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-04-25T19:59:24.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:24 smithi132 bash[18607]: cluster 2024-04-25T19:59:23.339473+0000 mon.a (mon.0) 375 : cluster [WRN] Health check update: Degraded data redundancy: 24/72 objects degraded (33.333%), 15 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-04-25T19:59:25.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:25 smithi132 bash[18607]: cluster 2024-04-25T19:59:23.847556+0000 mgr.x (mgr.14254) 16 : cluster [DBG] pgmap v12: 96 pgs: 15 active+undersized+degraded, 81 active+undersized; 65 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 617 B/s wr, 0 op/s; 24/72 objects degraded (33.333%) 2024-04-25T19:59:25.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:25 smithi132 bash[18607]: cluster 2024-04-25T19:59:23.847556+0000 mgr.x (mgr.14254) 16 : cluster [DBG] pgmap v12: 96 pgs: 15 active+undersized+degraded, 81 active+undersized; 65 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 617 B/s wr, 0 op/s; 24/72 objects degraded (33.333%) 2024-04-25T19:59:27.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:27 smithi132 bash[18607]: cluster 2024-04-25T19:59:25.849002+0000 mgr.x (mgr.14254) 17 : cluster [DBG] pgmap v13: 96 pgs: 15 active+undersized+degraded, 81 active+undersized; 65 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 561 B/s wr, 0 op/s; 24/72 objects degraded (33.333%) 2024-04-25T19:59:27.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:27 smithi132 bash[18607]: cluster 2024-04-25T19:59:25.849002+0000 mgr.x (mgr.14254) 17 : cluster [DBG] pgmap v13: 96 pgs: 15 active+undersized+degraded, 81 active+undersized; 65 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 561 B/s wr, 0 op/s; 24/72 objects degraded (33.333%) 2024-04-25T19:59:29.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:29 smithi132 bash[18607]: cluster 2024-04-25T19:59:27.850858+0000 mgr.x (mgr.14254) 18 : cluster [DBG] pgmap v14: 96 pgs: 15 active+undersized+degraded, 81 active+undersized; 65 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 511 B/s wr, 0 op/s; 24/72 objects degraded (33.333%) 2024-04-25T19:59:29.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:29 smithi132 bash[18607]: cluster 2024-04-25T19:59:27.850858+0000 mgr.x (mgr.14254) 18 : cluster [DBG] pgmap v14: 96 pgs: 15 active+undersized+degraded, 81 active+undersized; 65 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 511 B/s wr, 0 op/s; 24/72 objects degraded (33.333%) 2024-04-25T19:59:31.522 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:31 smithi132 bash[18607]: cluster 2024-04-25T19:59:29.852423+0000 mgr.x (mgr.14254) 19 : cluster [DBG] pgmap v15: 96 pgs: 15 active+undersized+degraded, 81 active+undersized; 65 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 511 B/s wr, 0 op/s; 24/72 objects degraded (33.333%) 2024-04-25T19:59:31.522 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:31 smithi132 bash[18607]: cluster 2024-04-25T19:59:29.852423+0000 mgr.x (mgr.14254) 19 : cluster [DBG] pgmap v15: 96 pgs: 15 active+undersized+degraded, 81 active+undersized; 65 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 511 B/s wr, 0 op/s; 24/72 objects degraded (33.333%) 2024-04-25T19:59:33.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:33 smithi132 bash[18607]: cluster 2024-04-25T19:59:31.854040+0000 mgr.x (mgr.14254) 20 : cluster [DBG] pgmap v16: 96 pgs: 15 active+undersized+degraded, 81 active+undersized; 65 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 255 B/s wr, 0 op/s; 24/72 objects degraded (33.333%) 2024-04-25T19:59:33.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:33 smithi132 bash[18607]: cluster 2024-04-25T19:59:31.854040+0000 mgr.x (mgr.14254) 20 : cluster [DBG] pgmap v16: 96 pgs: 15 active+undersized+degraded, 81 active+undersized; 65 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 255 B/s wr, 0 op/s; 24/72 objects degraded (33.333%) 2024-04-25T19:59:35.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:35 smithi132 bash[18607]: cluster 2024-04-25T19:59:33.856039+0000 mgr.x (mgr.14254) 21 : cluster [DBG] pgmap v17: 96 pgs: 15 active+undersized+degraded, 81 active+undersized; 65 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 255 B/s wr, 0 op/s; 24/72 objects degraded (33.333%) 2024-04-25T19:59:35.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:35 smithi132 bash[18607]: cluster 2024-04-25T19:59:33.856039+0000 mgr.x (mgr.14254) 21 : cluster [DBG] pgmap v17: 96 pgs: 15 active+undersized+degraded, 81 active+undersized; 65 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 255 B/s wr, 0 op/s; 24/72 objects degraded (33.333%) 2024-04-25T19:59:37.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:37 smithi132 bash[18607]: cluster 2024-04-25T19:59:35.857522+0000 mgr.x (mgr.14254) 22 : cluster [DBG] pgmap v18: 96 pgs: 15 active+undersized+degraded, 81 active+undersized; 65 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 24/72 objects degraded (33.333%) 2024-04-25T19:59:37.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:37 smithi132 bash[18607]: cluster 2024-04-25T19:59:35.857522+0000 mgr.x (mgr.14254) 22 : cluster [DBG] pgmap v18: 96 pgs: 15 active+undersized+degraded, 81 active+undersized; 65 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 24/72 objects degraded (33.333%) 2024-04-25T19:59:39.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:39 smithi132 bash[18607]: cluster 2024-04-25T19:59:37.859279+0000 mgr.x (mgr.14254) 23 : cluster [DBG] pgmap v19: 96 pgs: 15 active+undersized+degraded, 81 active+undersized; 65 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 24/72 objects degraded (33.333%) 2024-04-25T19:59:39.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:39 smithi132 bash[18607]: cluster 2024-04-25T19:59:37.859279+0000 mgr.x (mgr.14254) 23 : cluster [DBG] pgmap v19: 96 pgs: 15 active+undersized+degraded, 81 active+undersized; 65 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 24/72 objects degraded (33.333%) 2024-04-25T19:59:41.291 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:41 smithi132 bash[18607]: cluster 2024-04-25T19:59:39.860747+0000 mgr.x (mgr.14254) 24 : cluster [DBG] pgmap v20: 96 pgs: 15 active+undersized+degraded, 81 active+undersized; 65 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 24/72 objects degraded (33.333%) 2024-04-25T19:59:41.291 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:41 smithi132 bash[18607]: cluster 2024-04-25T19:59:39.860747+0000 mgr.x (mgr.14254) 24 : cluster [DBG] pgmap v20: 96 pgs: 15 active+undersized+degraded, 81 active+undersized; 65 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 24/72 objects degraded (33.333%) 2024-04-25T19:59:41.931 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- bash -c 'ceph smb apply -i -' 2024-04-25T19:59:43.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:43 smithi132 bash[18607]: cluster 2024-04-25T19:59:41.862104+0000 mgr.x (mgr.14254) 25 : cluster [DBG] pgmap v21: 96 pgs: 15 active+undersized+degraded, 81 active+undersized; 65 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 24/72 objects degraded (33.333%) 2024-04-25T19:59:43.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:43 smithi132 bash[18607]: cluster 2024-04-25T19:59:41.862104+0000 mgr.x (mgr.14254) 25 : cluster [DBG] pgmap v21: 96 pgs: 15 active+undersized+degraded, 81 active+undersized; 65 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 24/72 objects degraded (33.333%) 2024-04-25T19:59:45.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:45 smithi132 bash[18607]: cluster 2024-04-25T19:59:43.864070+0000 mgr.x (mgr.14254) 26 : cluster [DBG] pgmap v22: 96 pgs: 15 active+undersized+degraded, 81 active+undersized; 65 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 24/72 objects degraded (33.333%) 2024-04-25T19:59:45.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:45 smithi132 bash[18607]: cluster 2024-04-25T19:59:43.864070+0000 mgr.x (mgr.14254) 26 : cluster [DBG] pgmap v22: 96 pgs: 15 active+undersized+degraded, 81 active+undersized; 65 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 24/72 objects degraded (33.333%) 2024-04-25T19:59:46.690 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:59:47.480 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:47 smithi132 bash[18607]: cluster 2024-04-25T19:59:45.865679+0000 mgr.x (mgr.14254) 27 : cluster [DBG] pgmap v23: 96 pgs: 15 active+undersized+degraded, 81 active+undersized; 65 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 24/72 objects degraded (33.333%) 2024-04-25T19:59:47.480 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:47 smithi132 bash[18607]: cluster 2024-04-25T19:59:45.865679+0000 mgr.x (mgr.14254) 27 : cluster [DBG] pgmap v23: 96 pgs: 15 active+undersized+degraded, 81 active+undersized; 65 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 24/72 objects degraded (33.333%) 2024-04-25T19:59:49.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:49 smithi132 bash[18607]: cluster 2024-04-25T19:59:47.867433+0000 mgr.x (mgr.14254) 28 : cluster [DBG] pgmap v24: 96 pgs: 15 active+undersized+degraded, 81 active+undersized; 65 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 24/72 objects degraded (33.333%) 2024-04-25T19:59:49.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:49 smithi132 bash[18607]: cluster 2024-04-25T19:59:47.867433+0000 mgr.x (mgr.14254) 28 : cluster [DBG] pgmap v24: 96 pgs: 15 active+undersized+degraded, 81 active+undersized; 65 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 24/72 objects degraded (33.333%) 2024-04-25T19:59:49.411 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-04-25T19:59:49.411 INFO:teuthology.orchestra.run.smithi132.stdout: "results": [ 2024-04-25T19:59:49.411 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-25T19:59:49.411 INFO:teuthology.orchestra.run.smithi132.stdout: "resource": { 2024-04-25T19:59:49.411 INFO:teuthology.orchestra.run.smithi132.stdout: "auth": { 2024-04-25T19:59:49.411 INFO:teuthology.orchestra.run.smithi132.stdout: "password": "Passw0rd", 2024-04-25T19:59:49.412 INFO:teuthology.orchestra.run.smithi132.stdout: "username": "Administrator" 2024-04-25T19:59:49.412 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-25T19:59:49.412 INFO:teuthology.orchestra.run.smithi132.stdout: "auth_id": "join1-admin", 2024-04-25T19:59:49.412 INFO:teuthology.orchestra.run.smithi132.stdout: "intent": "present", 2024-04-25T19:59:49.412 INFO:teuthology.orchestra.run.smithi132.stdout: "resource_type": "ceph.smb.join.auth" 2024-04-25T19:59:49.412 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-25T19:59:49.412 INFO:teuthology.orchestra.run.smithi132.stdout: "state": "created", 2024-04-25T19:59:49.412 INFO:teuthology.orchestra.run.smithi132.stdout: "success": true 2024-04-25T19:59:49.413 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-25T19:59:49.413 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-25T19:59:49.413 INFO:teuthology.orchestra.run.smithi132.stdout: "resource": { 2024-04-25T19:59:49.413 INFO:teuthology.orchestra.run.smithi132.stdout: "auth_mode": "active-directory", 2024-04-25T19:59:49.413 INFO:teuthology.orchestra.run.smithi132.stdout: "cluster_id": "modtest1", 2024-04-25T19:59:49.413 INFO:teuthology.orchestra.run.smithi132.stdout: "custom_dns": [ 2024-04-25T19:59:49.413 INFO:teuthology.orchestra.run.smithi132.stdout: "192.168.76.204" 2024-04-25T19:59:49.413 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-25T19:59:49.413 INFO:teuthology.orchestra.run.smithi132.stdout: "domain_settings": { 2024-04-25T19:59:49.414 INFO:teuthology.orchestra.run.smithi132.stdout: "join_sources": [ 2024-04-25T19:59:49.414 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-25T19:59:49.414 INFO:teuthology.orchestra.run.smithi132.stdout: "ref": "join1-admin", 2024-04-25T19:59:49.414 INFO:teuthology.orchestra.run.smithi132.stdout: "source_type": "resource" 2024-04-25T19:59:49.414 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-25T19:59:49.414 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-25T19:59:49.414 INFO:teuthology.orchestra.run.smithi132.stdout: "realm": "DOMAIN1.SINK.TEST" 2024-04-25T19:59:49.414 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-25T19:59:49.415 INFO:teuthology.orchestra.run.smithi132.stdout: "intent": "present", 2024-04-25T19:59:49.415 INFO:teuthology.orchestra.run.smithi132.stdout: "placement": { 2024-04-25T19:59:49.415 INFO:teuthology.orchestra.run.smithi132.stdout: "count": 1 2024-04-25T19:59:49.415 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-25T19:59:49.415 INFO:teuthology.orchestra.run.smithi132.stdout: "resource_type": "ceph.smb.cluster" 2024-04-25T19:59:49.415 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-25T19:59:49.415 INFO:teuthology.orchestra.run.smithi132.stdout: "state": "created", 2024-04-25T19:59:49.415 INFO:teuthology.orchestra.run.smithi132.stdout: "success": true 2024-04-25T19:59:49.416 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-25T19:59:49.416 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-25T19:59:49.416 INFO:teuthology.orchestra.run.smithi132.stdout: "resource": { 2024-04-25T19:59:49.416 INFO:teuthology.orchestra.run.smithi132.stdout: "browseable": true, 2024-04-25T19:59:49.416 INFO:teuthology.orchestra.run.smithi132.stdout: "cephfs": { 2024-04-25T19:59:49.416 INFO:teuthology.orchestra.run.smithi132.stdout: "path": "/", 2024-04-25T19:59:49.416 INFO:teuthology.orchestra.run.smithi132.stdout: "provider": "samba-vfs", 2024-04-25T19:59:49.416 INFO:teuthology.orchestra.run.smithi132.stdout: "subvolume": "sv1", 2024-04-25T19:59:49.416 INFO:teuthology.orchestra.run.smithi132.stdout: "subvolumegroup": "smb", 2024-04-25T19:59:49.417 INFO:teuthology.orchestra.run.smithi132.stdout: "volume": "cephfs" 2024-04-25T19:59:49.417 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-25T19:59:49.417 INFO:teuthology.orchestra.run.smithi132.stdout: "cluster_id": "modtest1", 2024-04-25T19:59:49.417 INFO:teuthology.orchestra.run.smithi132.stdout: "intent": "present", 2024-04-25T19:59:49.417 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "share1", 2024-04-25T19:59:49.417 INFO:teuthology.orchestra.run.smithi132.stdout: "readonly": false, 2024-04-25T19:59:49.417 INFO:teuthology.orchestra.run.smithi132.stdout: "resource_type": "ceph.smb.share", 2024-04-25T19:59:49.417 INFO:teuthology.orchestra.run.smithi132.stdout: "share_id": "share1" 2024-04-25T19:59:49.418 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-25T19:59:49.418 INFO:teuthology.orchestra.run.smithi132.stdout: "state": "created", 2024-04-25T19:59:49.418 INFO:teuthology.orchestra.run.smithi132.stdout: "success": true 2024-04-25T19:59:49.418 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-25T19:59:49.418 INFO:teuthology.orchestra.run.smithi132.stdout: { 2024-04-25T19:59:49.418 INFO:teuthology.orchestra.run.smithi132.stdout: "resource": { 2024-04-25T19:59:49.418 INFO:teuthology.orchestra.run.smithi132.stdout: "browseable": true, 2024-04-25T19:59:49.418 INFO:teuthology.orchestra.run.smithi132.stdout: "cephfs": { 2024-04-25T19:59:49.419 INFO:teuthology.orchestra.run.smithi132.stdout: "path": "/", 2024-04-25T19:59:49.419 INFO:teuthology.orchestra.run.smithi132.stdout: "provider": "samba-vfs", 2024-04-25T19:59:49.419 INFO:teuthology.orchestra.run.smithi132.stdout: "subvolume": "sv2", 2024-04-25T19:59:49.419 INFO:teuthology.orchestra.run.smithi132.stdout: "subvolumegroup": "smb", 2024-04-25T19:59:49.419 INFO:teuthology.orchestra.run.smithi132.stdout: "volume": "cephfs" 2024-04-25T19:59:49.419 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-25T19:59:49.419 INFO:teuthology.orchestra.run.smithi132.stdout: "cluster_id": "modtest1", 2024-04-25T19:59:49.419 INFO:teuthology.orchestra.run.smithi132.stdout: "intent": "present", 2024-04-25T19:59:49.420 INFO:teuthology.orchestra.run.smithi132.stdout: "name": "share2", 2024-04-25T19:59:49.420 INFO:teuthology.orchestra.run.smithi132.stdout: "readonly": false, 2024-04-25T19:59:49.420 INFO:teuthology.orchestra.run.smithi132.stdout: "resource_type": "ceph.smb.share", 2024-04-25T19:59:49.420 INFO:teuthology.orchestra.run.smithi132.stdout: "share_id": "share2" 2024-04-25T19:59:49.420 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-04-25T19:59:49.420 INFO:teuthology.orchestra.run.smithi132.stdout: "state": "created", 2024-04-25T19:59:49.420 INFO:teuthology.orchestra.run.smithi132.stdout: "success": true 2024-04-25T19:59:49.420 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-04-25T19:59:49.420 INFO:teuthology.orchestra.run.smithi132.stdout: ], 2024-04-25T19:59:49.421 INFO:teuthology.orchestra.run.smithi132.stdout: "success": true 2024-04-25T19:59:49.421 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-04-25T19:59:49.615 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:49 smithi132 bash[18860]: debug 2024-04-25T19:59:49.339+0000 7f90d8f2d700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:59:49.616 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:49 smithi132 bash[18860]: debug 2024-04-25T19:59:49.339+0000 7f90d8f2d700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:59:49.616 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:49 smithi132 bash[18860]: debug 2024-04-25T19:59:49.339+0000 7f90d8f2d700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:59:49.616 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:49 smithi132 bash[18860]: debug 2024-04-25T19:59:49.339+0000 7f90d8f2d700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:59:49.616 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:49 smithi132 bash[18860]: debug 2024-04-25T19:59:49.343+0000 7f90d8f2d700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:59:49.616 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:49 smithi132 bash[18860]: debug 2024-04-25T19:59:49.355+0000 7f90e5f87700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:59:49.616 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:49 smithi132 bash[18860]: debug 2024-04-25T19:59:49.355+0000 7f90e5f87700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:59:49.616 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:49 smithi132 bash[18860]: debug 2024-04-25T19:59:49.355+0000 7f90e5f87700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:59:49.616 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:49 smithi132 bash[18860]: debug 2024-04-25T19:59:49.355+0000 7f90e5f87700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:59:49.616 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:49 smithi132 bash[18860]: debug 2024-04-25T19:59:49.355+0000 7f90e5f87700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T19:59:50.127 INFO:teuthology.run_tasks:Running task cephadm.wait_for_service... 2024-04-25T19:59:50.137 INFO:tasks.cephadm:Waiting for ceph service smb.modtest1 to start (timeout 300)... 2024-04-25T19:59:50.137 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T19:59:50.429 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:50 smithi132 systemd[1]: /etc/systemd/system/ceph-3114c848-033d-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-25T19:59:50.429 INFO:journalctl@ceph.osd.0.smithi132.stdout:Apr 25 19:59:50 smithi132 systemd[1]: /etc/systemd/system/ceph-3114c848-033d-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-25T19:59:50.430 INFO:journalctl@ceph.osd.1.smithi132.stdout:Apr 25 19:59:50 smithi132 systemd[1]: /etc/systemd/system/ceph-3114c848-033d-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-25T19:59:50.431 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.306466+0000 mgr.x (mgr.14254) 29 : audit [DBG] from='client.14268 -' entity='client.admin' cmd=[{"prefix": "smb apply", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:59:50.431 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.306466+0000 mgr.x (mgr.14254) 29 : audit [DBG] from='client.14268 -' entity='client.admin' cmd=[{"prefix": "smb apply", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:59:50.431 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.327925+0000 mon.a (mon.0) 376 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T19:59:50.431 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.327925+0000 mon.a (mon.0) 376 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T19:59:50.431 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.333979+0000 mon.a (mon.0) 377 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T19:59:50.431 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.333979+0000 mon.a (mon.0) 377 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T19:59:50.431 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.336148+0000 mon.a (mon.0) 378 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "group_name": "smb"}]: dispatch 2024-04-25T19:59:50.431 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.336148+0000 mon.a (mon.0) 378 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "group_name": "smb"}]: dispatch 2024-04-25T19:59:50.431 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.337574+0000 mgr.x (mgr.14254) 30 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "group_name": "smb"}]: dispatch 2024-04-25T19:59:50.431 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.337574+0000 mgr.x (mgr.14254) 30 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "group_name": "smb"}]: dispatch 2024-04-25T19:59:50.431 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.365955+0000 mon.a (mon.0) 379 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T19:59:50.431 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.365955+0000 mon.a (mon.0) 379 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T19:59:50.431 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.366597+0000 mon.a (mon.0) 380 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv2", "group_name": "smb"}]: dispatch 2024-04-25T19:59:50.432 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.366597+0000 mon.a (mon.0) 380 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv2", "group_name": "smb"}]: dispatch 2024-04-25T19:59:50.432 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.367543+0000 mgr.x (mgr.14254) 31 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv2", "group_name": "smb"}]: dispatch 2024-04-25T19:59:50.432 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.367543+0000 mgr.x (mgr.14254) 31 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv2", "group_name": "smb"}]: dispatch 2024-04-25T19:59:50.432 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.374939+0000 mon.a (mon.0) 381 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T19:59:50.432 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.374939+0000 mon.a (mon.0) 381 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T19:59:50.432 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.380983+0000 mon.a (mon.0) 382 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T19:59:50.432 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.380983+0000 mon.a (mon.0) 382 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T19:59:50.432 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.381704+0000 mon.a (mon.0) 383 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "group_name": "smb"}]: dispatch 2024-04-25T19:59:50.432 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.381704+0000 mon.a (mon.0) 383 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "group_name": "smb"}]: dispatch 2024-04-25T19:59:50.432 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.382640+0000 mgr.x (mgr.14254) 32 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "group_name": "smb"}]: dispatch 2024-04-25T19:59:50.432 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.382640+0000 mgr.x (mgr.14254) 32 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "group_name": "smb"}]: dispatch 2024-04-25T19:59:50.432 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.386111+0000 mon.a (mon.0) 384 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv2", "group_name": "smb"}]: dispatch 2024-04-25T19:59:50.432 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.386111+0000 mon.a (mon.0) 384 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv2", "group_name": "smb"}]: dispatch 2024-04-25T19:59:50.432 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.387025+0000 mgr.x (mgr.14254) 33 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv2", "group_name": "smb"}]: dispatch 2024-04-25T19:59:50.433 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.387025+0000 mgr.x (mgr.14254) 33 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv2", "group_name": "smb"}]: dispatch 2024-04-25T19:59:50.433 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.391597+0000 mon.a (mon.0) 385 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "config-key dump", "key": "smb/config/"}]: dispatch 2024-04-25T19:59:50.433 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.391597+0000 mon.a (mon.0) 385 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "config-key dump", "key": "smb/config/"}]: dispatch 2024-04-25T19:59:50.433 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.400982+0000 mon.a (mon.0) 386 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "fs authorize", "filesystem": "cephfs", "entity": "client.smb.fs.cluster.modtest1", "caps": ["/", "rw"]}]: dispatch 2024-04-25T19:59:50.433 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.400982+0000 mon.a (mon.0) 386 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "fs authorize", "filesystem": "cephfs", "entity": "client.smb.fs.cluster.modtest1", "caps": ["/", "rw"]}]: dispatch 2024-04-25T19:59:50.433 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.401802+0000 mon.a (mon.0) 387 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd='[{"prefix": "fs authorize", "filesystem": "cephfs", "entity": "client.smb.fs.cluster.modtest1", "caps": ["/", "rw"]}]': finished 2024-04-25T19:59:50.433 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.401802+0000 mon.a (mon.0) 387 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd='[{"prefix": "fs authorize", "filesystem": "cephfs", "entity": "client.smb.fs.cluster.modtest1", "caps": ["/", "rw"]}]': finished 2024-04-25T19:59:50.433 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: cephadm 2024-04-25T19:59:49.408057+0000 mgr.x (mgr.14254) 34 : cephadm [INF] Saving service smb.modtest1 spec with placement count:1 2024-04-25T19:59:50.433 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: cephadm 2024-04-25T19:59:49.408057+0000 mgr.x (mgr.14254) 34 : cephadm [INF] Saving service smb.modtest1 spec with placement count:1 2024-04-25T19:59:50.433 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.410107+0000 mon.a (mon.0) 388 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T19:59:50.433 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.410107+0000 mon.a (mon.0) 388 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T19:59:50.433 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.423832+0000 mon.a (mon.0) 389 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:59:50.433 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.423832+0000 mon.a (mon.0) 389 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:59:50.433 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.425481+0000 mon.a (mon.0) 390 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:59:50.434 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.425481+0000 mon.a (mon.0) 390 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:59:50.434 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.426186+0000 mon.a (mon.0) 391 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:59:50.434 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.426186+0000 mon.a (mon.0) 391 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:59:50.434 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.429795+0000 mon.a (mon.0) 392 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T19:59:50.434 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.429795+0000 mon.a (mon.0) 392 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T19:59:50.434 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.432498+0000 mon.a (mon.0) 393 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.smb.config.modtest1.smithi132.jrsyig", "caps": ["mon", "allow r, allow command \"config-key get\" with \"key\" prefix \"smb/config/modtest1/\"", "osd", "allow r pool=.smb"]}]: dispatch 2024-04-25T19:59:50.434 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.432498+0000 mon.a (mon.0) 393 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.smb.config.modtest1.smithi132.jrsyig", "caps": ["mon", "allow r, allow command \"config-key get\" with \"key\" prefix \"smb/config/modtest1/\"", "osd", "allow r pool=.smb"]}]: dispatch 2024-04-25T19:59:50.434 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.433279+0000 mon.a (mon.0) 394 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "client.smb.config.modtest1.smithi132.jrsyig", "caps": ["mon", "allow r, allow command \"config-key get\" with \"key\" prefix \"smb/config/modtest1/\"", "osd", "allow r pool=.smb"]}]': finished 2024-04-25T19:59:50.434 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.433279+0000 mon.a (mon.0) 394 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "client.smb.config.modtest1.smithi132.jrsyig", "caps": ["mon", "allow r, allow command \"config-key get\" with \"key\" prefix \"smb/config/modtest1/\"", "osd", "allow r pool=.smb"]}]': finished 2024-04-25T19:59:50.434 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.435064+0000 mon.a (mon.0) 395 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.smb.fs.cluster.modtest1"}]: dispatch 2024-04-25T19:59:50.434 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.435064+0000 mon.a (mon.0) 395 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.smb.fs.cluster.modtest1"}]: dispatch 2024-04-25T19:59:50.434 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.435624+0000 mon.a (mon.0) 396 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:59:50.434 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 bash[18607]: audit 2024-04-25T19:59:49.435624+0000 mon.a (mon.0) 396 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:59:50.435 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 systemd[1]: /etc/systemd/system/ceph-3114c848-033d-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-25T19:59:50.687 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:50 smithi132 systemd[1]: /etc/systemd/system/ceph-3114c848-033d-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-25T19:59:50.688 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 19:59:50 smithi132 systemd[1]: /etc/systemd/system/ceph-3114c848-033d-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-25T19:59:50.688 INFO:journalctl@ceph.osd.0.smithi132.stdout:Apr 25 19:59:50 smithi132 systemd[1]: /etc/systemd/system/ceph-3114c848-033d-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-25T19:59:50.688 INFO:journalctl@ceph.osd.1.smithi132.stdout:Apr 25 19:59:50 smithi132 systemd[1]: /etc/systemd/system/ceph-3114c848-033d-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-25T19:59:51.721 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:51 smithi132 bash[18607]: cephadm 2024-04-25T19:59:49.436464+0000 mgr.x (mgr.14254) 35 : cephadm [INF] Deploying daemon smb.modtest1.smithi132.jrsyig on smithi132 2024-04-25T19:59:51.722 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:51 smithi132 bash[18607]: cephadm 2024-04-25T19:59:49.436464+0000 mgr.x (mgr.14254) 35 : cephadm [INF] Deploying daemon smb.modtest1.smithi132.jrsyig on smithi132 2024-04-25T19:59:51.722 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:51 smithi132 bash[18607]: cluster 2024-04-25T19:59:49.868065+0000 mgr.x (mgr.14254) 36 : cluster [DBG] pgmap v25: 96 pgs: 15 active+undersized+degraded, 81 active+undersized; 65 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 24/72 objects degraded (33.333%) 2024-04-25T19:59:51.722 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:51 smithi132 bash[18607]: cluster 2024-04-25T19:59:49.868065+0000 mgr.x (mgr.14254) 36 : cluster [DBG] pgmap v25: 96 pgs: 15 active+undersized+degraded, 81 active+undersized; 65 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 24/72 objects degraded (33.333%) 2024-04-25T19:59:51.722 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:51 smithi132 bash[18607]: cluster 2024-04-25T19:59:50.445103+0000 mon.a (mon.0) 397 : cluster [DBG] mgrmap e20: x(active, since 38s) 2024-04-25T19:59:51.722 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:51 smithi132 bash[18607]: cluster 2024-04-25T19:59:50.445103+0000 mon.a (mon.0) 397 : cluster [DBG] mgrmap e20: x(active, since 38s) 2024-04-25T19:59:53.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:53 smithi132 bash[18607]: cluster 2024-04-25T19:59:51.869463+0000 mgr.x (mgr.14254) 37 : cluster [DBG] pgmap v26: 96 pgs: 15 active+undersized+degraded, 81 active+undersized; 65 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 24/72 objects degraded (33.333%) 2024-04-25T19:59:53.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:53 smithi132 bash[18607]: cluster 2024-04-25T19:59:51.869463+0000 mgr.x (mgr.14254) 37 : cluster [DBG] pgmap v26: 96 pgs: 15 active+undersized+degraded, 81 active+undersized; 65 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 24/72 objects degraded (33.333%) 2024-04-25T19:59:54.568 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:54 smithi132 bash[18607]: cluster 2024-04-25T19:59:54.328677+0000 mon.a (mon.0) 398 : cluster [WRN] Health check update: Degraded data redundancy: 27/81 objects degraded (33.333%), 16 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-04-25T19:59:54.568 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:54 smithi132 bash[18607]: cluster 2024-04-25T19:59:54.328677+0000 mon.a (mon.0) 398 : cluster [WRN] Health check update: Degraded data redundancy: 27/81 objects degraded (33.333%), 16 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-04-25T19:59:54.887 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:59:55.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:55 smithi132 bash[18607]: cluster 2024-04-25T19:59:53.871549+0000 mgr.x (mgr.14254) 38 : cluster [DBG] pgmap v27: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 170 B/s rd, 682 B/s wr, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T19:59:55.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:55 smithi132 bash[18607]: cluster 2024-04-25T19:59:53.871549+0000 mgr.x (mgr.14254) 38 : cluster [DBG] pgmap v27: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 170 B/s rd, 682 B/s wr, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T19:59:57.731 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:57 smithi132 bash[18607]: cluster 2024-04-25T19:59:55.872734+0000 mgr.x (mgr.14254) 39 : cluster [DBG] pgmap v28: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 170 B/s rd, 682 B/s wr, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T19:59:57.731 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:57 smithi132 bash[18607]: cluster 2024-04-25T19:59:55.872734+0000 mgr.x (mgr.14254) 39 : cluster [DBG] pgmap v28: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 170 B/s rd, 682 B/s wr, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T19:59:59.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:59 smithi132 bash[18607]: cluster 2024-04-25T19:59:57.874448+0000 mgr.x (mgr.14254) 40 : cluster [DBG] pgmap v29: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 170 B/s rd, 682 B/s wr, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T19:59:59.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 19:59:59 smithi132 bash[18607]: cluster 2024-04-25T19:59:57.874448+0000 mgr.x (mgr.14254) 40 : cluster [DBG] pgmap v29: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 170 B/s rd, 682 B/s wr, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:00:01.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:01 smithi132 bash[18607]: cluster 2024-04-25T19:59:59.876330+0000 mgr.x (mgr.14254) 41 : cluster [DBG] pgmap v30: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 170 B/s rd, 682 B/s wr, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:00:01.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:01 smithi132 bash[18607]: cluster 2024-04-25T19:59:59.876330+0000 mgr.x (mgr.14254) 41 : cluster [DBG] pgmap v30: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 170 B/s rd, 682 B/s wr, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:00:03.727 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:03 smithi132 bash[18607]: cluster 2024-04-25T20:00:01.878202+0000 mgr.x (mgr.14254) 42 : cluster [DBG] pgmap v31: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 170 B/s rd, 682 B/s wr, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:00:03.728 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:03 smithi132 bash[18607]: cluster 2024-04-25T20:00:01.878202+0000 mgr.x (mgr.14254) 42 : cluster [DBG] pgmap v31: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 170 B/s rd, 682 B/s wr, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:00:04.600 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:00:04.600 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:00:05.732 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:05 smithi132 bash[18607]: cluster 2024-04-25T20:00:03.880180+0000 mgr.x (mgr.14254) 43 : cluster [DBG] pgmap v32: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 170 B/s rd, 682 B/s wr, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:00:05.732 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:05 smithi132 bash[18607]: cluster 2024-04-25T20:00:03.880180+0000 mgr.x (mgr.14254) 43 : cluster [DBG] pgmap v32: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 170 B/s rd, 682 B/s wr, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:00:06.013 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:00:06.636 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:06 smithi132 bash[18607]: audit 2024-04-25T20:00:04.600962+0000 mgr.x (mgr.14254) 44 : audit [DBG] from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:00:06.636 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:06 smithi132 bash[18607]: audit 2024-04-25T20:00:04.600962+0000 mgr.x (mgr.14254) 44 : audit [DBG] from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:00:07.014 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:00:07.649 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:07 smithi132 bash[18607]: cluster 2024-04-25T20:00:05.881685+0000 mgr.x (mgr.14254) 45 : cluster [DBG] pgmap v33: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:00:07.650 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:07 smithi132 bash[18607]: cluster 2024-04-25T20:00:05.881685+0000 mgr.x (mgr.14254) 45 : cluster [DBG] pgmap v33: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:00:09.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:09 smithi132 bash[18607]: cluster 2024-04-25T20:00:07.883610+0000 mgr.x (mgr.14254) 46 : cluster [DBG] pgmap v34: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 170 B/s rd, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:00:09.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:09 smithi132 bash[18607]: cluster 2024-04-25T20:00:07.883610+0000 mgr.x (mgr.14254) 46 : cluster [DBG] pgmap v34: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 170 B/s rd, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:00:09.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:09 smithi132 bash[18607]: audit 2024-04-25T20:00:08.790038+0000 mon.a (mon.0) 399 : audit [DBG] from='client.? 172.17.0.2:0/3177264433' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:00:09.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:09 smithi132 bash[18607]: audit 2024-04-25T20:00:08.790038+0000 mon.a (mon.0) 399 : audit [DBG] from='client.? 172.17.0.2:0/3177264433' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:00:11.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:11 smithi132 bash[18607]: cluster 2024-04-25T20:00:09.885111+0000 mgr.x (mgr.14254) 47 : cluster [DBG] pgmap v35: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 170 B/s rd, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:00:11.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:11 smithi132 bash[18607]: cluster 2024-04-25T20:00:09.885111+0000 mgr.x (mgr.14254) 47 : cluster [DBG] pgmap v35: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 170 B/s rd, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:00:12.766 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:00:13.670 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:13 smithi132 bash[18607]: cluster 2024-04-25T20:00:11.886617+0000 mgr.x (mgr.14254) 48 : cluster [DBG] pgmap v36: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 170 B/s rd, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:00:13.670 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:13 smithi132 bash[18607]: cluster 2024-04-25T20:00:11.886617+0000 mgr.x (mgr.14254) 48 : cluster [DBG] pgmap v36: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 170 B/s rd, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:00:15.549 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:00:15.549 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:00:15.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:15 smithi132 bash[18607]: cluster 2024-04-25T20:00:13.887572+0000 mgr.x (mgr.14254) 49 : cluster [DBG] pgmap v37: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 170 B/s rd, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:00:15.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:15 smithi132 bash[18607]: cluster 2024-04-25T20:00:13.887572+0000 mgr.x (mgr.14254) 49 : cluster [DBG] pgmap v37: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 170 B/s rd, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:00:16.231 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:00:16.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:16 smithi132 bash[18607]: audit 2024-04-25T20:00:15.548564+0000 mgr.x (mgr.14254) 50 : audit [DBG] from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:00:16.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:16 smithi132 bash[18607]: audit 2024-04-25T20:00:15.548564+0000 mgr.x (mgr.14254) 50 : audit [DBG] from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:00:16.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:16 smithi132 bash[18607]: cluster 2024-04-25T20:00:15.888985+0000 mgr.x (mgr.14254) 51 : cluster [DBG] pgmap v38: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 170 B/s rd, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:00:16.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:16 smithi132 bash[18607]: cluster 2024-04-25T20:00:15.888985+0000 mgr.x (mgr.14254) 51 : cluster [DBG] pgmap v38: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 170 B/s rd, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:00:16.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:16 smithi132 bash[18607]: cluster 2024-04-25T20:00:16.401910+0000 mon.a (mon.0) 400 : cluster [WRN] Health check update: Degraded data redundancy: 27/81 objects degraded (33.333%), 16 pgs degraded, 96 pgs undersized (PG_DEGRADED) 2024-04-25T20:00:16.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:16 smithi132 bash[18607]: cluster 2024-04-25T20:00:16.401910+0000 mon.a (mon.0) 400 : cluster [WRN] Health check update: Degraded data redundancy: 27/81 objects degraded (33.333%), 16 pgs degraded, 96 pgs undersized (PG_DEGRADED) 2024-04-25T20:00:17.232 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:00:19.088 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:18 smithi132 bash[18607]: cluster 2024-04-25T20:00:17.890752+0000 mgr.x (mgr.14254) 52 : cluster [DBG] pgmap v39: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 170 B/s rd, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:00:19.089 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:18 smithi132 bash[18607]: cluster 2024-04-25T20:00:17.890752+0000 mgr.x (mgr.14254) 52 : cluster [DBG] pgmap v39: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 170 B/s rd, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:00:21.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:20 smithi132 bash[18607]: cluster 2024-04-25T20:00:19.892158+0000 mgr.x (mgr.14254) 53 : cluster [DBG] pgmap v40: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:00:21.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:20 smithi132 bash[18607]: cluster 2024-04-25T20:00:19.892158+0000 mgr.x (mgr.14254) 53 : cluster [DBG] pgmap v40: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:00:21.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:20 smithi132 bash[18607]: audit 2024-04-25T20:00:20.108981+0000 mon.a (mon.0) 401 : audit [DBG] from='client.? 172.17.0.2:0/1791531650' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:00:21.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:20 smithi132 bash[18607]: audit 2024-04-25T20:00:20.108981+0000 mon.a (mon.0) 401 : audit [DBG] from='client.? 172.17.0.2:0/1791531650' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:00:21.992 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:00:23.183 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:22 smithi132 bash[18607]: cluster 2024-04-25T20:00:21.893558+0000 mgr.x (mgr.14254) 54 : cluster [DBG] pgmap v41: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:00:23.184 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:22 smithi132 bash[18607]: cluster 2024-04-25T20:00:21.893558+0000 mgr.x (mgr.14254) 54 : cluster [DBG] pgmap v41: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:00:24.789 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:00:24.789 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:00:25.269 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:24 smithi132 bash[18607]: cluster 2024-04-25T20:00:23.895367+0000 mgr.x (mgr.14254) 55 : cluster [DBG] pgmap v42: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:00:25.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:24 smithi132 bash[18607]: cluster 2024-04-25T20:00:23.895367+0000 mgr.x (mgr.14254) 55 : cluster [DBG] pgmap v42: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:00:25.437 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:00:26.269 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:25 smithi132 bash[18607]: audit 2024-04-25T20:00:24.792093+0000 mgr.x (mgr.14254) 56 : audit [DBG] from='client.14284 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:00:26.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:25 smithi132 bash[18607]: audit 2024-04-25T20:00:24.792093+0000 mgr.x (mgr.14254) 56 : audit [DBG] from='client.14284 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:00:26.438 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:00:27.269 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:26 smithi132 bash[18607]: cluster 2024-04-25T20:00:25.896796+0000 mgr.x (mgr.14254) 57 : cluster [DBG] pgmap v43: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:00:27.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:26 smithi132 bash[18607]: cluster 2024-04-25T20:00:25.896796+0000 mgr.x (mgr.14254) 57 : cluster [DBG] pgmap v43: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:00:29.269 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:28 smithi132 bash[18607]: cluster 2024-04-25T20:00:27.898551+0000 mgr.x (mgr.14254) 58 : cluster [DBG] pgmap v44: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:00:29.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:28 smithi132 bash[18607]: cluster 2024-04-25T20:00:27.898551+0000 mgr.x (mgr.14254) 58 : cluster [DBG] pgmap v44: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:00:31.192 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:00:31.214 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:30 smithi132 bash[18607]: cluster 2024-04-25T20:00:29.900004+0000 mgr.x (mgr.14254) 59 : cluster [DBG] pgmap v45: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:00:31.214 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:30 smithi132 bash[18607]: cluster 2024-04-25T20:00:29.900004+0000 mgr.x (mgr.14254) 59 : cluster [DBG] pgmap v45: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:00:32.269 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:31 smithi132 bash[18607]: audit 2024-04-25T20:00:31.440485+0000 mon.a (mon.0) 402 : audit [DBG] from='client.? 172.17.0.2:0/173158810' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:00:32.269 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:31 smithi132 bash[18607]: audit 2024-04-25T20:00:31.440485+0000 mon.a (mon.0) 402 : audit [DBG] from='client.? 172.17.0.2:0/173158810' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:00:33.269 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:32 smithi132 bash[18607]: cluster 2024-04-25T20:00:31.901259+0000 mgr.x (mgr.14254) 60 : cluster [DBG] pgmap v46: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:00:33.269 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:32 smithi132 bash[18607]: cluster 2024-04-25T20:00:31.901259+0000 mgr.x (mgr.14254) 60 : cluster [DBG] pgmap v46: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:00:33.980 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:00:33.981 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:00:34.599 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:00:35.269 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:34 smithi132 bash[18607]: cluster 2024-04-25T20:00:33.902303+0000 mgr.x (mgr.14254) 61 : cluster [DBG] pgmap v47: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:00:35.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:34 smithi132 bash[18607]: cluster 2024-04-25T20:00:33.902303+0000 mgr.x (mgr.14254) 61 : cluster [DBG] pgmap v47: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:00:35.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:34 smithi132 bash[18607]: audit 2024-04-25T20:00:33.982954+0000 mgr.x (mgr.14254) 62 : audit [DBG] from='client.14288 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:00:35.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:34 smithi132 bash[18607]: audit 2024-04-25T20:00:33.982954+0000 mgr.x (mgr.14254) 62 : audit [DBG] from='client.14288 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:00:35.600 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:00:37.269 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:36 smithi132 bash[18607]: cluster 2024-04-25T20:00:35.903698+0000 mgr.x (mgr.14254) 63 : cluster [DBG] pgmap v48: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:00:37.269 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:36 smithi132 bash[18607]: cluster 2024-04-25T20:00:35.903698+0000 mgr.x (mgr.14254) 63 : cluster [DBG] pgmap v48: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:00:39.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:39 smithi132 bash[18607]: cluster 2024-04-25T20:00:37.905499+0000 mgr.x (mgr.14254) 64 : cluster [DBG] pgmap v49: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:00:39.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:39 smithi132 bash[18607]: cluster 2024-04-25T20:00:37.905499+0000 mgr.x (mgr.14254) 64 : cluster [DBG] pgmap v49: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:00:40.360 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:00:41.474 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:41 smithi132 bash[18607]: cluster 2024-04-25T20:00:39.906953+0000 mgr.x (mgr.14254) 65 : cluster [DBG] pgmap v50: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:00:41.474 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:41 smithi132 bash[18607]: cluster 2024-04-25T20:00:39.906953+0000 mgr.x (mgr.14254) 65 : cluster [DBG] pgmap v50: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:00:43.067 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:00:43.067 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:00:43.706 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:43 smithi132 bash[18607]: cluster 2024-04-25T20:00:41.908152+0000 mgr.x (mgr.14254) 66 : cluster [DBG] pgmap v51: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:00:43.706 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:43 smithi132 bash[18607]: cluster 2024-04-25T20:00:41.908152+0000 mgr.x (mgr.14254) 66 : cluster [DBG] pgmap v51: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:00:43.706 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:43 smithi132 bash[18607]: audit 2024-04-25T20:00:42.764424+0000 mon.a (mon.0) 403 : audit [DBG] from='client.? 172.17.0.2:0/2784579710' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:00:43.706 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:43 smithi132 bash[18607]: audit 2024-04-25T20:00:42.764424+0000 mon.a (mon.0) 403 : audit [DBG] from='client.? 172.17.0.2:0/2784579710' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:00:43.707 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:00:44.708 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:00:44.719 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:44 smithi132 bash[18607]: audit 2024-04-25T20:00:43.067718+0000 mgr.x (mgr.14254) 67 : audit [DBG] from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:00:44.719 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:44 smithi132 bash[18607]: audit 2024-04-25T20:00:43.067718+0000 mgr.x (mgr.14254) 67 : audit [DBG] from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:00:45.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:45 smithi132 bash[18607]: cluster 2024-04-25T20:00:43.910042+0000 mgr.x (mgr.14254) 68 : cluster [DBG] pgmap v52: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:00:45.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:45 smithi132 bash[18607]: cluster 2024-04-25T20:00:43.910042+0000 mgr.x (mgr.14254) 68 : cluster [DBG] pgmap v52: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:00:46.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:46 smithi132 bash[18607]: cluster 2024-04-25T20:00:45.374407+0000 mon.a (mon.0) 404 : cluster [DBG] mgrmap e21: x(active, since 93s) 2024-04-25T20:00:46.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:46 smithi132 bash[18607]: cluster 2024-04-25T20:00:45.374407+0000 mon.a (mon.0) 404 : cluster [DBG] mgrmap e21: x(active, since 93s) 2024-04-25T20:00:47.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:47 smithi132 bash[18607]: cluster 2024-04-25T20:00:45.910990+0000 mgr.x (mgr.14254) 69 : cluster [DBG] pgmap v53: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:00:47.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:47 smithi132 bash[18607]: cluster 2024-04-25T20:00:45.910990+0000 mgr.x (mgr.14254) 69 : cluster [DBG] pgmap v53: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:00:49.469 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:00:49.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:49 smithi132 bash[18607]: cluster 2024-04-25T20:00:47.912793+0000 mgr.x (mgr.14254) 70 : cluster [DBG] pgmap v54: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 255 B/s wr, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:00:49.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:49 smithi132 bash[18607]: cluster 2024-04-25T20:00:47.912793+0000 mgr.x (mgr.14254) 70 : cluster [DBG] pgmap v54: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 255 B/s wr, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:00:51.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:51 smithi132 bash[18607]: cluster 2024-04-25T20:00:49.914176+0000 mgr.x (mgr.14254) 71 : cluster [DBG] pgmap v55: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 255 B/s wr, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:00:51.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:51 smithi132 bash[18607]: cluster 2024-04-25T20:00:49.914176+0000 mgr.x (mgr.14254) 71 : cluster [DBG] pgmap v55: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 255 B/s wr, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:00:52.176 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:00:52.176 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:00:52.824 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:00:53.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:53 smithi132 bash[18607]: cluster 2024-04-25T20:00:51.915074+0000 mgr.x (mgr.14254) 72 : cluster [DBG] pgmap v56: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 255 B/s wr, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:00:53.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:53 smithi132 bash[18607]: cluster 2024-04-25T20:00:51.915074+0000 mgr.x (mgr.14254) 72 : cluster [DBG] pgmap v56: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 255 B/s wr, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:00:53.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:53 smithi132 bash[18607]: audit 2024-04-25T20:00:52.177191+0000 mgr.x (mgr.14254) 73 : audit [DBG] from='client.14294 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:00:53.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:53 smithi132 bash[18607]: audit 2024-04-25T20:00:52.177191+0000 mgr.x (mgr.14254) 73 : audit [DBG] from='client.14294 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:00:53.825 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:00:54.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:54 smithi132 bash[18607]: audit 2024-04-25T20:00:54.090477+0000 mon.a (mon.0) 405 : audit [DBG] from='client.? 172.17.0.2:0/277865949' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:00:54.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:54 smithi132 bash[18607]: audit 2024-04-25T20:00:54.090477+0000 mon.a (mon.0) 405 : audit [DBG] from='client.? 172.17.0.2:0/277865949' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:00:55.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:55 smithi132 bash[18607]: cluster 2024-04-25T20:00:53.916858+0000 mgr.x (mgr.14254) 74 : cluster [DBG] pgmap v57: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 255 B/s wr, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:00:55.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:55 smithi132 bash[18607]: cluster 2024-04-25T20:00:53.916858+0000 mgr.x (mgr.14254) 74 : cluster [DBG] pgmap v57: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 255 B/s wr, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:00:57.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:57 smithi132 bash[18607]: cluster 2024-04-25T20:00:55.918222+0000 mgr.x (mgr.14254) 75 : cluster [DBG] pgmap v58: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 255 B/s wr, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:00:57.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:57 smithi132 bash[18607]: cluster 2024-04-25T20:00:55.918222+0000 mgr.x (mgr.14254) 75 : cluster [DBG] pgmap v58: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 255 B/s wr, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:00:58.589 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:00:59.671 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:59 smithi132 bash[18607]: cluster 2024-04-25T20:00:57.920291+0000 mgr.x (mgr.14254) 76 : cluster [DBG] pgmap v59: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 255 B/s wr, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:00:59.671 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:00:59 smithi132 bash[18607]: cluster 2024-04-25T20:00:57.920291+0000 mgr.x (mgr.14254) 76 : cluster [DBG] pgmap v59: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 255 B/s wr, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:01:01.189 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:01:01.189 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:01:01.519 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:01 smithi132 bash[18607]: cluster 2024-04-25T20:00:59.921761+0000 mgr.x (mgr.14254) 77 : cluster [DBG] pgmap v60: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:01.519 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:01 smithi132 bash[18607]: cluster 2024-04-25T20:00:59.921761+0000 mgr.x (mgr.14254) 77 : cluster [DBG] pgmap v60: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:01.849 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:01:02.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:02 smithi132 bash[18607]: audit 2024-04-25T20:01:01.191525+0000 mgr.x (mgr.14254) 78 : audit [DBG] from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:01:02.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:02 smithi132 bash[18607]: audit 2024-04-25T20:01:01.191525+0000 mgr.x (mgr.14254) 78 : audit [DBG] from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:01:02.850 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:01:03.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:03 smithi132 bash[18607]: cluster 2024-04-25T20:01:01.923172+0000 mgr.x (mgr.14254) 79 : cluster [DBG] pgmap v61: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:03.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:03 smithi132 bash[18607]: cluster 2024-04-25T20:01:01.923172+0000 mgr.x (mgr.14254) 79 : cluster [DBG] pgmap v61: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:05.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:05 smithi132 bash[18607]: cluster 2024-04-25T20:01:03.925020+0000 mgr.x (mgr.14254) 80 : cluster [DBG] pgmap v62: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:05.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:05 smithi132 bash[18607]: cluster 2024-04-25T20:01:03.925020+0000 mgr.x (mgr.14254) 80 : cluster [DBG] pgmap v62: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:06.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:06 smithi132 bash[18607]: audit 2024-04-25T20:01:05.422202+0000 mon.a (mon.0) 406 : audit [DBG] from='client.? 172.17.0.2:0/3087012734' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:01:06.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:06 smithi132 bash[18607]: audit 2024-04-25T20:01:05.422202+0000 mon.a (mon.0) 406 : audit [DBG] from='client.? 172.17.0.2:0/3087012734' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:01:07.606 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:01:07.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:07 smithi132 bash[18607]: cluster 2024-04-25T20:01:05.926971+0000 mgr.x (mgr.14254) 81 : cluster [DBG] pgmap v63: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:07.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:07 smithi132 bash[18607]: cluster 2024-04-25T20:01:05.926971+0000 mgr.x (mgr.14254) 81 : cluster [DBG] pgmap v63: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:09.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:09 smithi132 bash[18607]: cluster 2024-04-25T20:01:07.929623+0000 mgr.x (mgr.14254) 82 : cluster [DBG] pgmap v64: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:09.519 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:09 smithi132 bash[18607]: cluster 2024-04-25T20:01:07.929623+0000 mgr.x (mgr.14254) 82 : cluster [DBG] pgmap v64: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:10.314 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:01:10.315 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:01:11.007 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:01:11.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:11 smithi132 bash[18607]: cluster 2024-04-25T20:01:09.930555+0000 mgr.x (mgr.14254) 83 : cluster [DBG] pgmap v65: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:11.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:11 smithi132 bash[18607]: cluster 2024-04-25T20:01:09.930555+0000 mgr.x (mgr.14254) 83 : cluster [DBG] pgmap v65: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:11.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:11 smithi132 bash[18607]: audit 2024-04-25T20:01:10.315331+0000 mgr.x (mgr.14254) 84 : audit [DBG] from='client.14302 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:01:11.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:11 smithi132 bash[18607]: audit 2024-04-25T20:01:10.315331+0000 mgr.x (mgr.14254) 84 : audit [DBG] from='client.14302 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:01:12.008 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:01:13.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:13 smithi132 bash[18607]: cluster 2024-04-25T20:01:11.931999+0000 mgr.x (mgr.14254) 85 : cluster [DBG] pgmap v66: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:13.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:13 smithi132 bash[18607]: cluster 2024-04-25T20:01:11.931999+0000 mgr.x (mgr.14254) 85 : cluster [DBG] pgmap v66: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:15.725 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:15 smithi132 bash[18607]: cluster 2024-04-25T20:01:13.934117+0000 mgr.x (mgr.14254) 86 : cluster [DBG] pgmap v67: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:15.725 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:15 smithi132 bash[18607]: cluster 2024-04-25T20:01:13.934117+0000 mgr.x (mgr.14254) 86 : cluster [DBG] pgmap v67: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:16.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:16 smithi132 bash[18607]: cluster 2024-04-25T20:01:15.446140+0000 mon.a (mon.0) 407 : cluster [DBG] mgrmap e22: x(active, since 2m) 2024-04-25T20:01:16.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:16 smithi132 bash[18607]: cluster 2024-04-25T20:01:15.446140+0000 mon.a (mon.0) 407 : cluster [DBG] mgrmap e22: x(active, since 2m) 2024-04-25T20:01:16.769 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:01:17.758 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:17 smithi132 bash[18607]: cluster 2024-04-25T20:01:15.935702+0000 mgr.x (mgr.14254) 87 : cluster [DBG] pgmap v68: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:17.758 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:17 smithi132 bash[18607]: cluster 2024-04-25T20:01:15.935702+0000 mgr.x (mgr.14254) 87 : cluster [DBG] pgmap v68: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:17.758 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:17 smithi132 bash[18607]: audit 2024-04-25T20:01:16.753949+0000 mon.a (mon.0) 408 : audit [DBG] from='client.? 172.17.0.2:0/497146221' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:01:17.758 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:17 smithi132 bash[18607]: audit 2024-04-25T20:01:16.753949+0000 mon.a (mon.0) 408 : audit [DBG] from='client.? 172.17.0.2:0/497146221' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:01:18.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:18 smithi132 bash[18607]: cluster 2024-04-25T20:01:17.936791+0000 mgr.x (mgr.14254) 88 : cluster [DBG] pgmap v69: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 255 B/s wr, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:01:18.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:18 smithi132 bash[18607]: cluster 2024-04-25T20:01:17.936791+0000 mgr.x (mgr.14254) 88 : cluster [DBG] pgmap v69: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 255 B/s wr, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:01:19.724 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:01:19.724 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:01:20.348 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:01:21.268 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:20 smithi132 bash[18607]: audit 2024-04-25T20:01:19.725166+0000 mgr.x (mgr.14254) 89 : audit [DBG] from='client.14306 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:01:21.269 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:20 smithi132 bash[18607]: audit 2024-04-25T20:01:19.725166+0000 mgr.x (mgr.14254) 89 : audit [DBG] from='client.14306 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:01:21.269 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:20 smithi132 bash[18607]: cluster 2024-04-25T20:01:19.938171+0000 mgr.x (mgr.14254) 90 : cluster [DBG] pgmap v70: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 255 B/s wr, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:01:21.269 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:20 smithi132 bash[18607]: cluster 2024-04-25T20:01:19.938171+0000 mgr.x (mgr.14254) 90 : cluster [DBG] pgmap v70: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 255 B/s wr, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:01:21.350 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:01:23.268 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:22 smithi132 bash[18607]: cluster 2024-04-25T20:01:21.939611+0000 mgr.x (mgr.14254) 91 : cluster [DBG] pgmap v71: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 255 B/s wr, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:01:23.269 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:22 smithi132 bash[18607]: cluster 2024-04-25T20:01:21.939611+0000 mgr.x (mgr.14254) 91 : cluster [DBG] pgmap v71: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 255 B/s wr, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:01:25.268 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:24 smithi132 bash[18607]: cluster 2024-04-25T20:01:23.941525+0000 mgr.x (mgr.14254) 92 : cluster [DBG] pgmap v72: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 255 B/s wr, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:01:25.269 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:24 smithi132 bash[18607]: cluster 2024-04-25T20:01:23.941525+0000 mgr.x (mgr.14254) 92 : cluster [DBG] pgmap v72: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 255 B/s wr, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:01:26.108 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:01:27.057 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:26 smithi132 bash[18607]: cluster 2024-04-25T20:01:25.942986+0000 mgr.x (mgr.14254) 93 : cluster [DBG] pgmap v73: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 255 B/s wr, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:01:27.057 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:26 smithi132 bash[18607]: cluster 2024-04-25T20:01:25.942986+0000 mgr.x (mgr.14254) 93 : cluster [DBG] pgmap v73: 96 pgs: 16 active+undersized+degraded, 80 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 255 B/s wr, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:01:28.317 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:28 smithi132 bash[18607]: cluster 2024-04-25T20:01:27.998636+0000 mon.a (mon.0) 409 : cluster [WRN] Health check update: Degraded data redundancy: 27/81 objects degraded (33.333%), 17 pgs degraded, 96 pgs undersized (PG_DEGRADED) 2024-04-25T20:01:28.317 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:28 smithi132 bash[18607]: cluster 2024-04-25T20:01:27.998636+0000 mon.a (mon.0) 409 : cluster [WRN] Health check update: Degraded data redundancy: 27/81 objects degraded (33.333%), 17 pgs degraded, 96 pgs undersized (PG_DEGRADED) 2024-04-25T20:01:28.707 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:01:28.707 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:01:29.418 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:29 smithi132 bash[18607]: cluster 2024-04-25T20:01:27.944792+0000 mgr.x (mgr.14254) 94 : cluster [DBG] pgmap v74: 96 pgs: 17 active+undersized+degraded, 79 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 255 B/s wr, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:01:29.418 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:29 smithi132 bash[18607]: cluster 2024-04-25T20:01:27.944792+0000 mgr.x (mgr.14254) 94 : cluster [DBG] pgmap v74: 96 pgs: 17 active+undersized+degraded, 79 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 255 B/s wr, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:01:29.418 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:29 smithi132 bash[18607]: audit 2024-04-25T20:01:28.081320+0000 mon.a (mon.0) 410 : audit [DBG] from='client.? 172.17.0.2:0/3495446118' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:01:29.418 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:29 smithi132 bash[18607]: audit 2024-04-25T20:01:28.081320+0000 mon.a (mon.0) 410 : audit [DBG] from='client.? 172.17.0.2:0/3495446118' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:01:29.419 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:01:30.420 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:01:30.431 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:30 smithi132 bash[18607]: audit 2024-04-25T20:01:28.708170+0000 mgr.x (mgr.14254) 95 : audit [DBG] from='client.14310 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:01:30.431 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:30 smithi132 bash[18607]: audit 2024-04-25T20:01:28.708170+0000 mgr.x (mgr.14254) 95 : audit [DBG] from='client.14310 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:01:31.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:31 smithi132 bash[18607]: cluster 2024-04-25T20:01:29.946148+0000 mgr.x (mgr.14254) 96 : cluster [DBG] pgmap v75: 96 pgs: 17 active+undersized+degraded, 79 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:31.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:31 smithi132 bash[18607]: cluster 2024-04-25T20:01:29.946148+0000 mgr.x (mgr.14254) 96 : cluster [DBG] pgmap v75: 96 pgs: 17 active+undersized+degraded, 79 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:33.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:33 smithi132 bash[18607]: cluster 2024-04-25T20:01:31.947533+0000 mgr.x (mgr.14254) 97 : cluster [DBG] pgmap v76: 96 pgs: 17 active+undersized+degraded, 79 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:33.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:33 smithi132 bash[18607]: cluster 2024-04-25T20:01:31.947533+0000 mgr.x (mgr.14254) 97 : cluster [DBG] pgmap v76: 96 pgs: 17 active+undersized+degraded, 79 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:34.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:34 smithi132 bash[18607]: cluster 2024-04-25T20:01:34.061266+0000 mon.a (mon.0) 411 : cluster [WRN] Health check update: Degraded data redundancy: 27/81 objects degraded (33.333%), 18 pgs degraded, 96 pgs undersized (PG_DEGRADED) 2024-04-25T20:01:34.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:34 smithi132 bash[18607]: cluster 2024-04-25T20:01:34.061266+0000 mon.a (mon.0) 411 : cluster [WRN] Health check update: Degraded data redundancy: 27/81 objects degraded (33.333%), 18 pgs degraded, 96 pgs undersized (PG_DEGRADED) 2024-04-25T20:01:35.182 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:01:35.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:35 smithi132 bash[18607]: cluster 2024-04-25T20:01:33.949383+0000 mgr.x (mgr.14254) 98 : cluster [DBG] pgmap v77: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:35.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:35 smithi132 bash[18607]: cluster 2024-04-25T20:01:33.949383+0000 mgr.x (mgr.14254) 98 : cluster [DBG] pgmap v77: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:37.482 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:37 smithi132 bash[18607]: cluster 2024-04-25T20:01:35.950802+0000 mgr.x (mgr.14254) 99 : cluster [DBG] pgmap v78: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:37.482 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:37 smithi132 bash[18607]: cluster 2024-04-25T20:01:35.950802+0000 mgr.x (mgr.14254) 99 : cluster [DBG] pgmap v78: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:37.900 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:01:37.900 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:01:38.563 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:01:39.450 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:39 smithi132 bash[18607]: audit 2024-04-25T20:01:37.901379+0000 mgr.x (mgr.14254) 100 : audit [DBG] from='client.14312 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:01:39.450 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:39 smithi132 bash[18607]: audit 2024-04-25T20:01:37.901379+0000 mgr.x (mgr.14254) 100 : audit [DBG] from='client.14312 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:01:39.450 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:39 smithi132 bash[18607]: cluster 2024-04-25T20:01:37.951871+0000 mgr.x (mgr.14254) 101 : cluster [DBG] pgmap v79: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:39.450 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:39 smithi132 bash[18607]: cluster 2024-04-25T20:01:37.951871+0000 mgr.x (mgr.14254) 101 : cluster [DBG] pgmap v79: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:39.564 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:01:40.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:40 smithi132 bash[18607]: audit 2024-04-25T20:01:39.411871+0000 mon.a (mon.0) 412 : audit [DBG] from='client.? 172.17.0.2:0/3834403952' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:01:40.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:40 smithi132 bash[18607]: audit 2024-04-25T20:01:39.411871+0000 mon.a (mon.0) 412 : audit [DBG] from='client.? 172.17.0.2:0/3834403952' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:01:41.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:41 smithi132 bash[18607]: cluster 2024-04-25T20:01:39.953263+0000 mgr.x (mgr.14254) 102 : cluster [DBG] pgmap v80: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:41.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:41 smithi132 bash[18607]: cluster 2024-04-25T20:01:39.953263+0000 mgr.x (mgr.14254) 102 : cluster [DBG] pgmap v80: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:43.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:43 smithi132 bash[18607]: cluster 2024-04-25T20:01:41.954732+0000 mgr.x (mgr.14254) 103 : cluster [DBG] pgmap v81: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:43.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:43 smithi132 bash[18607]: cluster 2024-04-25T20:01:41.954732+0000 mgr.x (mgr.14254) 103 : cluster [DBG] pgmap v81: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:44.318 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:01:45.471 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:45 smithi132 bash[18607]: cluster 2024-04-25T20:01:43.956596+0000 mgr.x (mgr.14254) 104 : cluster [DBG] pgmap v82: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:45.472 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:45 smithi132 bash[18607]: cluster 2024-04-25T20:01:43.956596+0000 mgr.x (mgr.14254) 104 : cluster [DBG] pgmap v82: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:47.175 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:01:47.175 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:01:47.379 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:47 smithi132 bash[18607]: cluster 2024-04-25T20:01:45.957983+0000 mgr.x (mgr.14254) 105 : cluster [DBG] pgmap v83: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:47.380 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:47 smithi132 bash[18607]: cluster 2024-04-25T20:01:45.957983+0000 mgr.x (mgr.14254) 105 : cluster [DBG] pgmap v83: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:47.863 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:01:48.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:48 smithi132 bash[18607]: audit 2024-04-25T20:01:47.176882+0000 mgr.x (mgr.14254) 106 : audit [DBG] from='client.14316 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:01:48.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:48 smithi132 bash[18607]: audit 2024-04-25T20:01:47.176882+0000 mgr.x (mgr.14254) 106 : audit [DBG] from='client.14316 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:01:48.866 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:01:49.464 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:49 smithi132 bash[18607]: cluster 2024-04-25T20:01:47.959761+0000 mgr.x (mgr.14254) 107 : cluster [DBG] pgmap v84: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:49.464 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:49 smithi132 bash[18607]: cluster 2024-04-25T20:01:47.959761+0000 mgr.x (mgr.14254) 107 : cluster [DBG] pgmap v84: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:51.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:51 smithi132 bash[18607]: cluster 2024-04-25T20:01:49.961493+0000 mgr.x (mgr.14254) 108 : cluster [DBG] pgmap v85: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:51.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:51 smithi132 bash[18607]: cluster 2024-04-25T20:01:49.961493+0000 mgr.x (mgr.14254) 108 : cluster [DBG] pgmap v85: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:51.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:51 smithi132 bash[18607]: audit 2024-04-25T20:01:50.746763+0000 mon.a (mon.0) 413 : audit [DBG] from='client.? 172.17.0.2:0/107959378' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:01:51.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:51 smithi132 bash[18607]: audit 2024-04-25T20:01:50.746763+0000 mon.a (mon.0) 413 : audit [DBG] from='client.? 172.17.0.2:0/107959378' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:01:53.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:53 smithi132 bash[18607]: cluster 2024-04-25T20:01:51.962894+0000 mgr.x (mgr.14254) 109 : cluster [DBG] pgmap v86: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:53.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:53 smithi132 bash[18607]: cluster 2024-04-25T20:01:51.962894+0000 mgr.x (mgr.14254) 109 : cluster [DBG] pgmap v86: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:53.631 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:01:55.399 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:55 smithi132 bash[18607]: cluster 2024-04-25T20:01:53.964766+0000 mgr.x (mgr.14254) 110 : cluster [DBG] pgmap v87: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:55.399 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:55 smithi132 bash[18607]: cluster 2024-04-25T20:01:53.964766+0000 mgr.x (mgr.14254) 110 : cluster [DBG] pgmap v87: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:56.477 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:01:56.477 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:01:57.166 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:01:57.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:57 smithi132 bash[18607]: cluster 2024-04-25T20:01:55.966171+0000 mgr.x (mgr.14254) 111 : cluster [DBG] pgmap v88: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:57.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:57 smithi132 bash[18607]: cluster 2024-04-25T20:01:55.966171+0000 mgr.x (mgr.14254) 111 : cluster [DBG] pgmap v88: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:58.167 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:01:58.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:58 smithi132 bash[18607]: audit 2024-04-25T20:01:56.478686+0000 mgr.x (mgr.14254) 112 : audit [DBG] from='client.14320 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:01:58.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:58 smithi132 bash[18607]: audit 2024-04-25T20:01:56.478686+0000 mgr.x (mgr.14254) 112 : audit [DBG] from='client.14320 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:01:59.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:59 smithi132 bash[18607]: cluster 2024-04-25T20:01:57.967975+0000 mgr.x (mgr.14254) 113 : cluster [DBG] pgmap v89: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:01:59.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:01:59 smithi132 bash[18607]: cluster 2024-04-25T20:01:57.967975+0000 mgr.x (mgr.14254) 113 : cluster [DBG] pgmap v89: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:01.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:01 smithi132 bash[18607]: cluster 2024-04-25T20:01:59.969523+0000 mgr.x (mgr.14254) 114 : cluster [DBG] pgmap v90: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:01.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:01 smithi132 bash[18607]: cluster 2024-04-25T20:01:59.969523+0000 mgr.x (mgr.14254) 114 : cluster [DBG] pgmap v90: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:02.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:02 smithi132 bash[18607]: audit 2024-04-25T20:02:02.082052+0000 mon.a (mon.0) 414 : audit [DBG] from='client.? 172.17.0.2:0/3687008916' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:02:02.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:02 smithi132 bash[18607]: audit 2024-04-25T20:02:02.082052+0000 mon.a (mon.0) 414 : audit [DBG] from='client.? 172.17.0.2:0/3687008916' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:02:02.927 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:02:03.268 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:03 smithi132 bash[18607]: cluster 2024-04-25T20:02:01.970961+0000 mgr.x (mgr.14254) 115 : cluster [DBG] pgmap v91: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:03.268 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:03 smithi132 bash[18607]: cluster 2024-04-25T20:02:01.970961+0000 mgr.x (mgr.14254) 115 : cluster [DBG] pgmap v91: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:05.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:05 smithi132 bash[18607]: cluster 2024-04-25T20:02:03.972090+0000 mgr.x (mgr.14254) 116 : cluster [DBG] pgmap v92: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:05.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:05 smithi132 bash[18607]: cluster 2024-04-25T20:02:03.972090+0000 mgr.x (mgr.14254) 116 : cluster [DBG] pgmap v92: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:05.646 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:02:05.646 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:02:06.303 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:02:07.304 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:02:07.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:07 smithi132 bash[18607]: audit 2024-04-25T20:02:05.648371+0000 mgr.x (mgr.14254) 117 : audit [DBG] from='client.14324 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:02:07.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:07 smithi132 bash[18607]: audit 2024-04-25T20:02:05.648371+0000 mgr.x (mgr.14254) 117 : audit [DBG] from='client.14324 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:02:07.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:07 smithi132 bash[18607]: cluster 2024-04-25T20:02:05.973637+0000 mgr.x (mgr.14254) 118 : cluster [DBG] pgmap v93: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:07.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:07 smithi132 bash[18607]: cluster 2024-04-25T20:02:05.973637+0000 mgr.x (mgr.14254) 118 : cluster [DBG] pgmap v93: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:09.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:09 smithi132 bash[18607]: cluster 2024-04-25T20:02:07.975464+0000 mgr.x (mgr.14254) 119 : cluster [DBG] pgmap v94: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:09.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:09 smithi132 bash[18607]: cluster 2024-04-25T20:02:07.975464+0000 mgr.x (mgr.14254) 119 : cluster [DBG] pgmap v94: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:11.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:11 smithi132 bash[18607]: cluster 2024-04-25T20:02:09.976905+0000 mgr.x (mgr.14254) 120 : cluster [DBG] pgmap v95: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:11.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:11 smithi132 bash[18607]: cluster 2024-04-25T20:02:09.976905+0000 mgr.x (mgr.14254) 120 : cluster [DBG] pgmap v95: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:12.062 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:02:13.267 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:13 smithi132 bash[18607]: cluster 2024-04-25T20:02:11.978779+0000 mgr.x (mgr.14254) 121 : cluster [DBG] pgmap v96: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:13.268 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:13 smithi132 bash[18607]: cluster 2024-04-25T20:02:11.978779+0000 mgr.x (mgr.14254) 121 : cluster [DBG] pgmap v96: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:14.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:14 smithi132 bash[18607]: audit 2024-04-25T20:02:13.415396+0000 mon.a (mon.0) 415 : audit [DBG] from='client.? 172.17.0.2:0/3648703809' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:02:14.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:14 smithi132 bash[18607]: audit 2024-04-25T20:02:13.415396+0000 mon.a (mon.0) 415 : audit [DBG] from='client.? 172.17.0.2:0/3648703809' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:02:15.142 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:02:15.142 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:02:15.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:15 smithi132 bash[18607]: cluster 2024-04-25T20:02:13.980899+0000 mgr.x (mgr.14254) 122 : cluster [DBG] pgmap v97: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:15.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:15 smithi132 bash[18607]: cluster 2024-04-25T20:02:13.980899+0000 mgr.x (mgr.14254) 122 : cluster [DBG] pgmap v97: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:15.851 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:02:16.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:16 smithi132 bash[18607]: audit 2024-04-25T20:02:15.144322+0000 mgr.x (mgr.14254) 123 : audit [DBG] from='client.14328 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:02:16.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:16 smithi132 bash[18607]: audit 2024-04-25T20:02:15.144322+0000 mgr.x (mgr.14254) 123 : audit [DBG] from='client.14328 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:02:16.852 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:02:17.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:17 smithi132 bash[18607]: cluster 2024-04-25T20:02:15.982436+0000 mgr.x (mgr.14254) 124 : cluster [DBG] pgmap v98: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:17.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:17 smithi132 bash[18607]: cluster 2024-04-25T20:02:15.982436+0000 mgr.x (mgr.14254) 124 : cluster [DBG] pgmap v98: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:19.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:19 smithi132 bash[18607]: cluster 2024-04-25T20:02:17.984351+0000 mgr.x (mgr.14254) 125 : cluster [DBG] pgmap v99: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:19.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:19 smithi132 bash[18607]: cluster 2024-04-25T20:02:17.984351+0000 mgr.x (mgr.14254) 125 : cluster [DBG] pgmap v99: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:21.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:21 smithi132 bash[18607]: cluster 2024-04-25T20:02:19.985853+0000 mgr.x (mgr.14254) 126 : cluster [DBG] pgmap v100: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:21.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:21 smithi132 bash[18607]: cluster 2024-04-25T20:02:19.985853+0000 mgr.x (mgr.14254) 126 : cluster [DBG] pgmap v100: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:21.612 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:02:23.466 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:23 smithi132 bash[18607]: cluster 2024-04-25T20:02:21.987660+0000 mgr.x (mgr.14254) 127 : cluster [DBG] pgmap v101: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:23.466 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:23 smithi132 bash[18607]: cluster 2024-04-25T20:02:21.987660+0000 mgr.x (mgr.14254) 127 : cluster [DBG] pgmap v101: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:24.585 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:02:24.585 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:02:25.216 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:02:25.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:25 smithi132 bash[18607]: cluster 2024-04-25T20:02:23.989557+0000 mgr.x (mgr.14254) 128 : cluster [DBG] pgmap v102: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:25.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:25 smithi132 bash[18607]: cluster 2024-04-25T20:02:23.989557+0000 mgr.x (mgr.14254) 128 : cluster [DBG] pgmap v102: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:26.217 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:02:26.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:26 smithi132 bash[18607]: audit 2024-04-25T20:02:24.590704+0000 mgr.x (mgr.14254) 129 : audit [DBG] from='client.14330 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:02:26.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:26 smithi132 bash[18607]: audit 2024-04-25T20:02:24.590704+0000 mgr.x (mgr.14254) 129 : audit [DBG] from='client.14330 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:02:27.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:27 smithi132 bash[18607]: cluster 2024-04-25T20:02:25.990962+0000 mgr.x (mgr.14254) 130 : cluster [DBG] pgmap v103: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:27.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:27 smithi132 bash[18607]: cluster 2024-04-25T20:02:25.990962+0000 mgr.x (mgr.14254) 130 : cluster [DBG] pgmap v103: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:29.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:29 smithi132 bash[18607]: cluster 2024-04-25T20:02:27.992758+0000 mgr.x (mgr.14254) 131 : cluster [DBG] pgmap v104: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:29.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:29 smithi132 bash[18607]: cluster 2024-04-25T20:02:27.992758+0000 mgr.x (mgr.14254) 131 : cluster [DBG] pgmap v104: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:29.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:29 smithi132 bash[18607]: audit 2024-04-25T20:02:28.749162+0000 mon.a (mon.0) 416 : audit [DBG] from='client.? 172.17.0.2:0/1111715874' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:02:29.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:29 smithi132 bash[18607]: audit 2024-04-25T20:02:28.749162+0000 mon.a (mon.0) 416 : audit [DBG] from='client.? 172.17.0.2:0/1111715874' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:02:30.977 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:02:31.267 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:31 smithi132 bash[18607]: cluster 2024-04-25T20:02:29.994378+0000 mgr.x (mgr.14254) 132 : cluster [DBG] pgmap v105: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:31.267 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:31 smithi132 bash[18607]: cluster 2024-04-25T20:02:29.994378+0000 mgr.x (mgr.14254) 132 : cluster [DBG] pgmap v105: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:33.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:33 smithi132 bash[18607]: cluster 2024-04-25T20:02:31.995386+0000 mgr.x (mgr.14254) 133 : cluster [DBG] pgmap v106: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:33.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:33 smithi132 bash[18607]: cluster 2024-04-25T20:02:31.995386+0000 mgr.x (mgr.14254) 133 : cluster [DBG] pgmap v106: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:33.681 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:02:33.682 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:02:34.366 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:02:35.367 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:02:35.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:35 smithi132 bash[18607]: audit 2024-04-25T20:02:33.683336+0000 mgr.x (mgr.14254) 134 : audit [DBG] from='client.14334 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:02:35.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:35 smithi132 bash[18607]: audit 2024-04-25T20:02:33.683336+0000 mgr.x (mgr.14254) 134 : audit [DBG] from='client.14334 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:02:35.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:35 smithi132 bash[18607]: cluster 2024-04-25T20:02:33.997425+0000 mgr.x (mgr.14254) 135 : cluster [DBG] pgmap v107: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:35.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:35 smithi132 bash[18607]: cluster 2024-04-25T20:02:33.997425+0000 mgr.x (mgr.14254) 135 : cluster [DBG] pgmap v107: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:37.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:37 smithi132 bash[18607]: cluster 2024-04-25T20:02:35.998975+0000 mgr.x (mgr.14254) 136 : cluster [DBG] pgmap v108: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:37.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:37 smithi132 bash[18607]: cluster 2024-04-25T20:02:35.998975+0000 mgr.x (mgr.14254) 136 : cluster [DBG] pgmap v108: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:39.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:39 smithi132 bash[18607]: cluster 2024-04-25T20:02:38.000809+0000 mgr.x (mgr.14254) 137 : cluster [DBG] pgmap v109: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:39.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:39 smithi132 bash[18607]: cluster 2024-04-25T20:02:38.000809+0000 mgr.x (mgr.14254) 137 : cluster [DBG] pgmap v109: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:40.123 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:02:41.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:41 smithi132 bash[18607]: cluster 2024-04-25T20:02:40.002150+0000 mgr.x (mgr.14254) 138 : cluster [DBG] pgmap v110: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:41.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:41 smithi132 bash[18607]: cluster 2024-04-25T20:02:40.002150+0000 mgr.x (mgr.14254) 138 : cluster [DBG] pgmap v110: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:43.069 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:02:43.069 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:02:43.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:43 smithi132 bash[18607]: cluster 2024-04-25T20:02:42.003601+0000 mgr.x (mgr.14254) 139 : cluster [DBG] pgmap v111: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:43.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:43 smithi132 bash[18607]: cluster 2024-04-25T20:02:42.003601+0000 mgr.x (mgr.14254) 139 : cluster [DBG] pgmap v111: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:43.776 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:02:44.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:44 smithi132 bash[18607]: audit 2024-04-25T20:02:43.071867+0000 mgr.x (mgr.14254) 140 : audit [DBG] from='client.14336 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:02:44.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:44 smithi132 bash[18607]: audit 2024-04-25T20:02:43.071867+0000 mgr.x (mgr.14254) 140 : audit [DBG] from='client.14336 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:02:44.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:44 smithi132 bash[18607]: audit 2024-04-25T20:02:44.080120+0000 mon.a (mon.0) 417 : audit [DBG] from='client.? 172.17.0.2:0/4147313400' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:02:44.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:44 smithi132 bash[18607]: audit 2024-04-25T20:02:44.080120+0000 mon.a (mon.0) 417 : audit [DBG] from='client.? 172.17.0.2:0/4147313400' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:02:44.777 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:02:45.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:45 smithi132 bash[18607]: cluster 2024-04-25T20:02:44.005470+0000 mgr.x (mgr.14254) 141 : cluster [DBG] pgmap v112: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:45.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:45 smithi132 bash[18607]: cluster 2024-04-25T20:02:44.005470+0000 mgr.x (mgr.14254) 141 : cluster [DBG] pgmap v112: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:47.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:47 smithi132 bash[18607]: cluster 2024-04-25T20:02:46.006652+0000 mgr.x (mgr.14254) 142 : cluster [DBG] pgmap v113: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:47.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:47 smithi132 bash[18607]: cluster 2024-04-25T20:02:46.006652+0000 mgr.x (mgr.14254) 142 : cluster [DBG] pgmap v113: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:49.538 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:02:49.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:49 smithi132 bash[18607]: cluster 2024-04-25T20:02:48.008441+0000 mgr.x (mgr.14254) 143 : cluster [DBG] pgmap v114: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:49.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:49 smithi132 bash[18607]: cluster 2024-04-25T20:02:48.008441+0000 mgr.x (mgr.14254) 143 : cluster [DBG] pgmap v114: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:51.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:51 smithi132 bash[18607]: cluster 2024-04-25T20:02:50.009841+0000 mgr.x (mgr.14254) 144 : cluster [DBG] pgmap v115: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:51.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:51 smithi132 bash[18607]: cluster 2024-04-25T20:02:50.009841+0000 mgr.x (mgr.14254) 144 : cluster [DBG] pgmap v115: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:52.234 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:02:52.235 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:02:52.926 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:02:53.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:53 smithi132 bash[18607]: cluster 2024-04-25T20:02:52.010458+0000 mgr.x (mgr.14254) 145 : cluster [DBG] pgmap v116: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:53.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:53 smithi132 bash[18607]: cluster 2024-04-25T20:02:52.010458+0000 mgr.x (mgr.14254) 145 : cluster [DBG] pgmap v116: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:53.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:53 smithi132 bash[18607]: audit 2024-04-25T20:02:52.237418+0000 mgr.x (mgr.14254) 146 : audit [DBG] from='client.14340 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:02:53.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:53 smithi132 bash[18607]: audit 2024-04-25T20:02:52.237418+0000 mgr.x (mgr.14254) 146 : audit [DBG] from='client.14340 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:02:53.927 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:02:55.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:55 smithi132 bash[18607]: cluster 2024-04-25T20:02:54.012194+0000 mgr.x (mgr.14254) 147 : cluster [DBG] pgmap v117: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:55.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:55 smithi132 bash[18607]: cluster 2024-04-25T20:02:54.012194+0000 mgr.x (mgr.14254) 147 : cluster [DBG] pgmap v117: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:57.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:57 smithi132 bash[18607]: cluster 2024-04-25T20:02:56.013569+0000 mgr.x (mgr.14254) 148 : cluster [DBG] pgmap v118: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:57.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:57 smithi132 bash[18607]: cluster 2024-04-25T20:02:56.013569+0000 mgr.x (mgr.14254) 148 : cluster [DBG] pgmap v118: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:58.685 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:02:59.677 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:59 smithi132 bash[18607]: cluster 2024-04-25T20:02:58.015389+0000 mgr.x (mgr.14254) 149 : cluster [DBG] pgmap v119: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:02:59.677 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:02:59 smithi132 bash[18607]: cluster 2024-04-25T20:02:58.015389+0000 mgr.x (mgr.14254) 149 : cluster [DBG] pgmap v119: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:00.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:00 smithi132 bash[18607]: audit 2024-04-25T20:02:59.418634+0000 mon.a (mon.0) 418 : audit [DBG] from='client.? 172.17.0.2:0/1631899484' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:03:00.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:00 smithi132 bash[18607]: audit 2024-04-25T20:02:59.418634+0000 mon.a (mon.0) 418 : audit [DBG] from='client.? 172.17.0.2:0/1631899484' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:03:01.572 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:03:01.572 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:03:01.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:01 smithi132 bash[18607]: cluster 2024-04-25T20:03:00.016797+0000 mgr.x (mgr.14254) 150 : cluster [DBG] pgmap v120: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:01.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:01 smithi132 bash[18607]: cluster 2024-04-25T20:03:00.016797+0000 mgr.x (mgr.14254) 150 : cluster [DBG] pgmap v120: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:02.243 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:03:03.245 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:03:03.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:03 smithi132 bash[18607]: audit 2024-04-25T20:03:01.574583+0000 mgr.x (mgr.14254) 151 : audit [DBG] from='client.14344 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:03:03.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:03 smithi132 bash[18607]: audit 2024-04-25T20:03:01.574583+0000 mgr.x (mgr.14254) 151 : audit [DBG] from='client.14344 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:03:03.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:03 smithi132 bash[18607]: cluster 2024-04-25T20:03:02.018166+0000 mgr.x (mgr.14254) 152 : cluster [DBG] pgmap v121: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:03.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:03 smithi132 bash[18607]: cluster 2024-04-25T20:03:02.018166+0000 mgr.x (mgr.14254) 152 : cluster [DBG] pgmap v121: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:05.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:05 smithi132 bash[18607]: cluster 2024-04-25T20:03:04.019938+0000 mgr.x (mgr.14254) 153 : cluster [DBG] pgmap v122: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:05.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:05 smithi132 bash[18607]: cluster 2024-04-25T20:03:04.019938+0000 mgr.x (mgr.14254) 153 : cluster [DBG] pgmap v122: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:07.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:07 smithi132 bash[18607]: cluster 2024-04-25T20:03:06.020747+0000 mgr.x (mgr.14254) 154 : cluster [DBG] pgmap v123: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:07.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:07 smithi132 bash[18607]: cluster 2024-04-25T20:03:06.020747+0000 mgr.x (mgr.14254) 154 : cluster [DBG] pgmap v123: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:08.006 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:03:09.469 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:09 smithi132 bash[18607]: cluster 2024-04-25T20:03:08.021854+0000 mgr.x (mgr.14254) 155 : cluster [DBG] pgmap v124: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:09.469 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:09 smithi132 bash[18607]: cluster 2024-04-25T20:03:08.021854+0000 mgr.x (mgr.14254) 155 : cluster [DBG] pgmap v124: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:10.912 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:03:10.913 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:03:11.559 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:03:11.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:11 smithi132 bash[18607]: cluster 2024-04-25T20:03:10.022645+0000 mgr.x (mgr.14254) 156 : cluster [DBG] pgmap v125: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:11.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:11 smithi132 bash[18607]: cluster 2024-04-25T20:03:10.022645+0000 mgr.x (mgr.14254) 156 : cluster [DBG] pgmap v125: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:12.560 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:03:12.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:12 smithi132 bash[18607]: audit 2024-04-25T20:03:10.915061+0000 mgr.x (mgr.14254) 157 : audit [DBG] from='client.14346 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:03:12.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:12 smithi132 bash[18607]: audit 2024-04-25T20:03:10.915061+0000 mgr.x (mgr.14254) 157 : audit [DBG] from='client.14346 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:03:13.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:13 smithi132 bash[18607]: cluster 2024-04-25T20:03:12.024002+0000 mgr.x (mgr.14254) 158 : cluster [DBG] pgmap v126: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:13.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:13 smithi132 bash[18607]: cluster 2024-04-25T20:03:12.024002+0000 mgr.x (mgr.14254) 158 : cluster [DBG] pgmap v126: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:15.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:15 smithi132 bash[18607]: cluster 2024-04-25T20:03:14.025829+0000 mgr.x (mgr.14254) 159 : cluster [DBG] pgmap v127: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:15.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:15 smithi132 bash[18607]: cluster 2024-04-25T20:03:14.025829+0000 mgr.x (mgr.14254) 159 : cluster [DBG] pgmap v127: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:15.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:15 smithi132 bash[18607]: audit 2024-04-25T20:03:14.756068+0000 mon.a (mon.0) 419 : audit [DBG] from='client.? 172.17.0.2:0/4083403014' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:03:15.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:15 smithi132 bash[18607]: audit 2024-04-25T20:03:14.756068+0000 mon.a (mon.0) 419 : audit [DBG] from='client.? 172.17.0.2:0/4083403014' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:03:17.312 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:03:17.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:17 smithi132 bash[18607]: cluster 2024-04-25T20:03:16.027220+0000 mgr.x (mgr.14254) 160 : cluster [DBG] pgmap v128: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:17.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:17 smithi132 bash[18607]: cluster 2024-04-25T20:03:16.027220+0000 mgr.x (mgr.14254) 160 : cluster [DBG] pgmap v128: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:19.698 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:19 smithi132 bash[18607]: cluster 2024-04-25T20:03:18.028883+0000 mgr.x (mgr.14254) 161 : cluster [DBG] pgmap v129: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:19.698 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:19 smithi132 bash[18607]: cluster 2024-04-25T20:03:18.028883+0000 mgr.x (mgr.14254) 161 : cluster [DBG] pgmap v129: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:20.120 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:03:20.120 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:03:20.797 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:03:21.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:21 smithi132 bash[18607]: cluster 2024-04-25T20:03:20.030224+0000 mgr.x (mgr.14254) 162 : cluster [DBG] pgmap v130: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:21.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:21 smithi132 bash[18607]: cluster 2024-04-25T20:03:20.030224+0000 mgr.x (mgr.14254) 162 : cluster [DBG] pgmap v130: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:21.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:21 smithi132 bash[18607]: audit 2024-04-25T20:03:20.123346+0000 mgr.x (mgr.14254) 163 : audit [DBG] from='client.14350 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:03:21.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:21 smithi132 bash[18607]: audit 2024-04-25T20:03:20.123346+0000 mgr.x (mgr.14254) 163 : audit [DBG] from='client.14350 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:03:21.798 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:03:23.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:23 smithi132 bash[18607]: cluster 2024-04-25T20:03:22.030986+0000 mgr.x (mgr.14254) 164 : cluster [DBG] pgmap v131: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:23.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:23 smithi132 bash[18607]: cluster 2024-04-25T20:03:22.030986+0000 mgr.x (mgr.14254) 164 : cluster [DBG] pgmap v131: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:25.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:25 smithi132 bash[18607]: cluster 2024-04-25T20:03:24.032947+0000 mgr.x (mgr.14254) 165 : cluster [DBG] pgmap v132: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:25.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:25 smithi132 bash[18607]: cluster 2024-04-25T20:03:24.032947+0000 mgr.x (mgr.14254) 165 : cluster [DBG] pgmap v132: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:26.564 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:03:27.714 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:27 smithi132 bash[18607]: cluster 2024-04-25T20:03:26.034334+0000 mgr.x (mgr.14254) 166 : cluster [DBG] pgmap v133: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:27.715 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:27 smithi132 bash[18607]: cluster 2024-04-25T20:03:26.034334+0000 mgr.x (mgr.14254) 166 : cluster [DBG] pgmap v133: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:29.220 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:03:29.220 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:03:29.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:29 smithi132 bash[18607]: cluster 2024-04-25T20:03:28.036099+0000 mgr.x (mgr.14254) 167 : cluster [DBG] pgmap v134: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:29.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:29 smithi132 bash[18607]: cluster 2024-04-25T20:03:28.036099+0000 mgr.x (mgr.14254) 167 : cluster [DBG] pgmap v134: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:29.833 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:03:30.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:30 smithi132 bash[18607]: audit 2024-04-25T20:03:29.226420+0000 mgr.x (mgr.14254) 168 : audit [DBG] from='client.14352 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:03:30.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:30 smithi132 bash[18607]: audit 2024-04-25T20:03:29.226420+0000 mgr.x (mgr.14254) 168 : audit [DBG] from='client.14352 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:03:30.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:30 smithi132 bash[18607]: audit 2024-04-25T20:03:30.099119+0000 mon.a (mon.0) 420 : audit [DBG] from='client.? 172.17.0.2:0/1648411652' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:03:30.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:30 smithi132 bash[18607]: audit 2024-04-25T20:03:30.099119+0000 mon.a (mon.0) 420 : audit [DBG] from='client.? 172.17.0.2:0/1648411652' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:03:30.834 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:03:31.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:31 smithi132 bash[18607]: cluster 2024-04-25T20:03:30.037525+0000 mgr.x (mgr.14254) 169 : cluster [DBG] pgmap v135: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:31.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:31 smithi132 bash[18607]: cluster 2024-04-25T20:03:30.037525+0000 mgr.x (mgr.14254) 169 : cluster [DBG] pgmap v135: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:33.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:33 smithi132 bash[18607]: cluster 2024-04-25T20:03:32.039564+0000 mgr.x (mgr.14254) 170 : cluster [DBG] pgmap v136: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:33.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:33 smithi132 bash[18607]: cluster 2024-04-25T20:03:32.039564+0000 mgr.x (mgr.14254) 170 : cluster [DBG] pgmap v136: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:35.599 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:03:35.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:35 smithi132 bash[18607]: cluster 2024-04-25T20:03:34.041648+0000 mgr.x (mgr.14254) 171 : cluster [DBG] pgmap v137: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:35.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:35 smithi132 bash[18607]: cluster 2024-04-25T20:03:34.041648+0000 mgr.x (mgr.14254) 171 : cluster [DBG] pgmap v137: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:37.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:37 smithi132 bash[18607]: cluster 2024-04-25T20:03:36.043175+0000 mgr.x (mgr.14254) 172 : cluster [DBG] pgmap v138: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:37.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:37 smithi132 bash[18607]: cluster 2024-04-25T20:03:36.043175+0000 mgr.x (mgr.14254) 172 : cluster [DBG] pgmap v138: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:38.323 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:03:38.323 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:03:38.973 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:03:39.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:39 smithi132 bash[18607]: cluster 2024-04-25T20:03:38.044144+0000 mgr.x (mgr.14254) 173 : cluster [DBG] pgmap v139: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:39.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:39 smithi132 bash[18607]: cluster 2024-04-25T20:03:38.044144+0000 mgr.x (mgr.14254) 173 : cluster [DBG] pgmap v139: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:39.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:39 smithi132 bash[18607]: audit 2024-04-25T20:03:38.326767+0000 mgr.x (mgr.14254) 174 : audit [DBG] from='client.14356 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:03:39.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:39 smithi132 bash[18607]: audit 2024-04-25T20:03:38.326767+0000 mgr.x (mgr.14254) 174 : audit [DBG] from='client.14356 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:03:39.974 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:03:41.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:41 smithi132 bash[18607]: cluster 2024-04-25T20:03:40.045758+0000 mgr.x (mgr.14254) 175 : cluster [DBG] pgmap v140: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:41.786 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:41 smithi132 bash[18607]: cluster 2024-04-25T20:03:40.045758+0000 mgr.x (mgr.14254) 175 : cluster [DBG] pgmap v140: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:43.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:43 smithi132 bash[18607]: cluster 2024-04-25T20:03:42.047167+0000 mgr.x (mgr.14254) 176 : cluster [DBG] pgmap v141: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:43.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:43 smithi132 bash[18607]: cluster 2024-04-25T20:03:42.047167+0000 mgr.x (mgr.14254) 176 : cluster [DBG] pgmap v141: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:44.731 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:03:45.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:45 smithi132 bash[18607]: cluster 2024-04-25T20:03:44.049163+0000 mgr.x (mgr.14254) 177 : cluster [DBG] pgmap v142: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:45.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:45 smithi132 bash[18607]: cluster 2024-04-25T20:03:44.049163+0000 mgr.x (mgr.14254) 177 : cluster [DBG] pgmap v142: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:45.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:45 smithi132 bash[18607]: audit 2024-04-25T20:03:45.433398+0000 mon.a (mon.0) 421 : audit [DBG] from='client.? 172.17.0.2:0/4273996304' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:03:45.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:45 smithi132 bash[18607]: audit 2024-04-25T20:03:45.433398+0000 mon.a (mon.0) 421 : audit [DBG] from='client.? 172.17.0.2:0/4273996304' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:03:47.477 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:03:47.477 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:03:47.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:47 smithi132 bash[18607]: cluster 2024-04-25T20:03:46.050736+0000 mgr.x (mgr.14254) 178 : cluster [DBG] pgmap v143: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:47.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:47 smithi132 bash[18607]: cluster 2024-04-25T20:03:46.050736+0000 mgr.x (mgr.14254) 178 : cluster [DBG] pgmap v143: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:48.126 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:03:49.128 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:03:49.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:49 smithi132 bash[18607]: audit 2024-04-25T20:03:47.483111+0000 mgr.x (mgr.14254) 179 : audit [DBG] from='client.14360 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:03:49.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:49 smithi132 bash[18607]: audit 2024-04-25T20:03:47.483111+0000 mgr.x (mgr.14254) 179 : audit [DBG] from='client.14360 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:03:49.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:49 smithi132 bash[18607]: cluster 2024-04-25T20:03:48.052563+0000 mgr.x (mgr.14254) 180 : cluster [DBG] pgmap v144: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:49.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:49 smithi132 bash[18607]: cluster 2024-04-25T20:03:48.052563+0000 mgr.x (mgr.14254) 180 : cluster [DBG] pgmap v144: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:50.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:50 smithi132 bash[18607]: cluster 2024-04-25T20:03:50.054010+0000 mgr.x (mgr.14254) 181 : cluster [DBG] pgmap v145: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:50.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:50 smithi132 bash[18607]: cluster 2024-04-25T20:03:50.054010+0000 mgr.x (mgr.14254) 181 : cluster [DBG] pgmap v145: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:53.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:53 smithi132 bash[18607]: cluster 2024-04-25T20:03:52.055430+0000 mgr.x (mgr.14254) 182 : cluster [DBG] pgmap v146: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:53.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:53 smithi132 bash[18607]: cluster 2024-04-25T20:03:52.055430+0000 mgr.x (mgr.14254) 182 : cluster [DBG] pgmap v146: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:53.889 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:03:55.462 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:55 smithi132 bash[18607]: cluster 2024-04-25T20:03:54.057360+0000 mgr.x (mgr.14254) 183 : cluster [DBG] pgmap v147: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:55.462 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:55 smithi132 bash[18607]: cluster 2024-04-25T20:03:54.057360+0000 mgr.x (mgr.14254) 183 : cluster [DBG] pgmap v147: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:56.636 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:03:56.636 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:03:57.297 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:03:57.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:57 smithi132 bash[18607]: cluster 2024-04-25T20:03:56.058777+0000 mgr.x (mgr.14254) 184 : cluster [DBG] pgmap v148: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:57.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:57 smithi132 bash[18607]: cluster 2024-04-25T20:03:56.058777+0000 mgr.x (mgr.14254) 184 : cluster [DBG] pgmap v148: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:58.298 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:03:58.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:58 smithi132 bash[18607]: audit 2024-04-25T20:03:56.639835+0000 mgr.x (mgr.14254) 185 : audit [DBG] from='client.14362 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:03:58.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:58 smithi132 bash[18607]: audit 2024-04-25T20:03:56.639835+0000 mgr.x (mgr.14254) 185 : audit [DBG] from='client.14362 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:03:59.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:59 smithi132 bash[18607]: cluster 2024-04-25T20:03:58.060550+0000 mgr.x (mgr.14254) 186 : cluster [DBG] pgmap v149: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:03:59.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:03:59 smithi132 bash[18607]: cluster 2024-04-25T20:03:58.060550+0000 mgr.x (mgr.14254) 186 : cluster [DBG] pgmap v149: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:01.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:01 smithi132 bash[18607]: cluster 2024-04-25T20:04:00.061922+0000 mgr.x (mgr.14254) 187 : cluster [DBG] pgmap v150: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:01.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:01 smithi132 bash[18607]: cluster 2024-04-25T20:04:00.061922+0000 mgr.x (mgr.14254) 187 : cluster [DBG] pgmap v150: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:01.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:01 smithi132 bash[18607]: audit 2024-04-25T20:04:00.783319+0000 mon.a (mon.0) 422 : audit [DBG] from='client.? 172.17.0.2:0/3617553109' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:04:01.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:01 smithi132 bash[18607]: audit 2024-04-25T20:04:00.783319+0000 mon.a (mon.0) 422 : audit [DBG] from='client.? 172.17.0.2:0/3617553109' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:04:03.055 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:04:03.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:03 smithi132 bash[18607]: cluster 2024-04-25T20:04:02.063305+0000 mgr.x (mgr.14254) 188 : cluster [DBG] pgmap v151: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:03.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:03 smithi132 bash[18607]: cluster 2024-04-25T20:04:02.063305+0000 mgr.x (mgr.14254) 188 : cluster [DBG] pgmap v151: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:05.392 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:05 smithi132 bash[18607]: cluster 2024-04-25T20:04:04.064195+0000 mgr.x (mgr.14254) 189 : cluster [DBG] pgmap v152: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:05.392 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:05 smithi132 bash[18607]: cluster 2024-04-25T20:04:04.064195+0000 mgr.x (mgr.14254) 189 : cluster [DBG] pgmap v152: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:05.796 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:04:05.796 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:04:06.493 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:04:07.494 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:04:07.506 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:07 smithi132 bash[18607]: audit 2024-04-25T20:04:05.800147+0000 mgr.x (mgr.14254) 190 : audit [DBG] from='client.14366 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:04:07.506 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:07 smithi132 bash[18607]: audit 2024-04-25T20:04:05.800147+0000 mgr.x (mgr.14254) 190 : audit [DBG] from='client.14366 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:04:07.506 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:07 smithi132 bash[18607]: cluster 2024-04-25T20:04:06.065608+0000 mgr.x (mgr.14254) 191 : cluster [DBG] pgmap v153: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:07.506 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:07 smithi132 bash[18607]: cluster 2024-04-25T20:04:06.065608+0000 mgr.x (mgr.14254) 191 : cluster [DBG] pgmap v153: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:09.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:09 smithi132 bash[18607]: cluster 2024-04-25T20:04:08.067348+0000 mgr.x (mgr.14254) 192 : cluster [DBG] pgmap v154: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:09.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:09 smithi132 bash[18607]: cluster 2024-04-25T20:04:08.067348+0000 mgr.x (mgr.14254) 192 : cluster [DBG] pgmap v154: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:11.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:11 smithi132 bash[18607]: cluster 2024-04-25T20:04:10.068799+0000 mgr.x (mgr.14254) 193 : cluster [DBG] pgmap v155: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:11.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:11 smithi132 bash[18607]: cluster 2024-04-25T20:04:10.068799+0000 mgr.x (mgr.14254) 193 : cluster [DBG] pgmap v155: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:12.254 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:04:13.398 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:13 smithi132 bash[18607]: cluster 2024-04-25T20:04:12.070127+0000 mgr.x (mgr.14254) 194 : cluster [DBG] pgmap v156: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:13.398 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:13 smithi132 bash[18607]: cluster 2024-04-25T20:04:12.070127+0000 mgr.x (mgr.14254) 194 : cluster [DBG] pgmap v156: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:14.917 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:04:14.917 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:04:15.265 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:15 smithi132 bash[18607]: cluster 2024-04-25T20:04:14.072127+0000 mgr.x (mgr.14254) 195 : cluster [DBG] pgmap v157: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:15.266 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:15 smithi132 bash[18607]: cluster 2024-04-25T20:04:14.072127+0000 mgr.x (mgr.14254) 195 : cluster [DBG] pgmap v157: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:15.645 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:04:16.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:16 smithi132 bash[18607]: audit 2024-04-25T20:04:14.921264+0000 mgr.x (mgr.14254) 196 : audit [DBG] from='client.14368 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:04:16.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:16 smithi132 bash[18607]: audit 2024-04-25T20:04:14.921264+0000 mgr.x (mgr.14254) 196 : audit [DBG] from='client.14368 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:04:16.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:16 smithi132 bash[18607]: audit 2024-04-25T20:04:16.118255+0000 mon.a (mon.0) 423 : audit [DBG] from='client.? 172.17.0.2:0/1593318099' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:04:16.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:16 smithi132 bash[18607]: audit 2024-04-25T20:04:16.118255+0000 mon.a (mon.0) 423 : audit [DBG] from='client.? 172.17.0.2:0/1593318099' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:04:16.646 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:04:17.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:17 smithi132 bash[18607]: cluster 2024-04-25T20:04:16.073514+0000 mgr.x (mgr.14254) 197 : cluster [DBG] pgmap v158: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:17.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:17 smithi132 bash[18607]: cluster 2024-04-25T20:04:16.073514+0000 mgr.x (mgr.14254) 197 : cluster [DBG] pgmap v158: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:19.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:19 smithi132 bash[18607]: cluster 2024-04-25T20:04:18.075426+0000 mgr.x (mgr.14254) 198 : cluster [DBG] pgmap v159: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:19.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:19 smithi132 bash[18607]: cluster 2024-04-25T20:04:18.075426+0000 mgr.x (mgr.14254) 198 : cluster [DBG] pgmap v159: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:21.405 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:04:21.427 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:21 smithi132 bash[18607]: cluster 2024-04-25T20:04:20.077008+0000 mgr.x (mgr.14254) 199 : cluster [DBG] pgmap v160: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:21.428 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:21 smithi132 bash[18607]: cluster 2024-04-25T20:04:20.077008+0000 mgr.x (mgr.14254) 199 : cluster [DBG] pgmap v160: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:23.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:23 smithi132 bash[18607]: cluster 2024-04-25T20:04:22.078478+0000 mgr.x (mgr.14254) 200 : cluster [DBG] pgmap v161: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:23.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:23 smithi132 bash[18607]: cluster 2024-04-25T20:04:22.078478+0000 mgr.x (mgr.14254) 200 : cluster [DBG] pgmap v161: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:24.215 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:04:24.216 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:04:24.855 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:04:25.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:25 smithi132 bash[18607]: cluster 2024-04-25T20:04:24.080440+0000 mgr.x (mgr.14254) 201 : cluster [DBG] pgmap v162: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:25.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:25 smithi132 bash[18607]: cluster 2024-04-25T20:04:24.080440+0000 mgr.x (mgr.14254) 201 : cluster [DBG] pgmap v162: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:25.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:25 smithi132 bash[18607]: audit 2024-04-25T20:04:24.219893+0000 mgr.x (mgr.14254) 202 : audit [DBG] from='client.14372 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:04:25.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:25 smithi132 bash[18607]: audit 2024-04-25T20:04:24.219893+0000 mgr.x (mgr.14254) 202 : audit [DBG] from='client.14372 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:04:25.859 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:04:27.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:27 smithi132 bash[18607]: cluster 2024-04-25T20:04:26.081287+0000 mgr.x (mgr.14254) 203 : cluster [DBG] pgmap v163: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:27.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:27 smithi132 bash[18607]: cluster 2024-04-25T20:04:26.081287+0000 mgr.x (mgr.14254) 203 : cluster [DBG] pgmap v163: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:29.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:29 smithi132 bash[18607]: cluster 2024-04-25T20:04:28.083015+0000 mgr.x (mgr.14254) 204 : cluster [DBG] pgmap v164: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:29.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:29 smithi132 bash[18607]: cluster 2024-04-25T20:04:28.083015+0000 mgr.x (mgr.14254) 204 : cluster [DBG] pgmap v164: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:30.626 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:04:31.494 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:31 smithi132 bash[18607]: cluster 2024-04-25T20:04:30.084407+0000 mgr.x (mgr.14254) 205 : cluster [DBG] pgmap v165: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:31.494 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:31 smithi132 bash[18607]: cluster 2024-04-25T20:04:30.084407+0000 mgr.x (mgr.14254) 205 : cluster [DBG] pgmap v165: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:32.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:32 smithi132 bash[18607]: audit 2024-04-25T20:04:31.454795+0000 mon.a (mon.0) 424 : audit [DBG] from='client.? 172.17.0.2:0/707097309' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:04:32.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:32 smithi132 bash[18607]: audit 2024-04-25T20:04:31.454795+0000 mon.a (mon.0) 424 : audit [DBG] from='client.? 172.17.0.2:0/707097309' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:04:33.489 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:04:33.489 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:04:33.502 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:33 smithi132 bash[18607]: cluster 2024-04-25T20:04:32.085835+0000 mgr.x (mgr.14254) 206 : cluster [DBG] pgmap v166: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:33.502 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:33 smithi132 bash[18607]: cluster 2024-04-25T20:04:32.085835+0000 mgr.x (mgr.14254) 206 : cluster [DBG] pgmap v166: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:34.148 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:04:35.148 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:04:35.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:35 smithi132 bash[18607]: audit 2024-04-25T20:04:33.493371+0000 mgr.x (mgr.14254) 207 : audit [DBG] from='client.14376 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:04:35.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:35 smithi132 bash[18607]: audit 2024-04-25T20:04:33.493371+0000 mgr.x (mgr.14254) 207 : audit [DBG] from='client.14376 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:04:35.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:35 smithi132 bash[18607]: cluster 2024-04-25T20:04:34.087775+0000 mgr.x (mgr.14254) 208 : cluster [DBG] pgmap v167: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:35.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:35 smithi132 bash[18607]: cluster 2024-04-25T20:04:34.087775+0000 mgr.x (mgr.14254) 208 : cluster [DBG] pgmap v167: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:37.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:37 smithi132 bash[18607]: cluster 2024-04-25T20:04:36.089351+0000 mgr.x (mgr.14254) 209 : cluster [DBG] pgmap v168: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:37.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:37 smithi132 bash[18607]: cluster 2024-04-25T20:04:36.089351+0000 mgr.x (mgr.14254) 209 : cluster [DBG] pgmap v168: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:39.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:39 smithi132 bash[18607]: cluster 2024-04-25T20:04:38.091177+0000 mgr.x (mgr.14254) 210 : cluster [DBG] pgmap v169: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:39.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:39 smithi132 bash[18607]: cluster 2024-04-25T20:04:38.091177+0000 mgr.x (mgr.14254) 210 : cluster [DBG] pgmap v169: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:39.900 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:04:41.759 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:41 smithi132 bash[18607]: cluster 2024-04-25T20:04:40.092541+0000 mgr.x (mgr.14254) 211 : cluster [DBG] pgmap v170: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:41.759 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:41 smithi132 bash[18607]: cluster 2024-04-25T20:04:40.092541+0000 mgr.x (mgr.14254) 211 : cluster [DBG] pgmap v170: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:42.924 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:04:42.925 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:04:43.544 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:04:43.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:43 smithi132 bash[18607]: cluster 2024-04-25T20:04:42.093899+0000 mgr.x (mgr.14254) 212 : cluster [DBG] pgmap v171: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:43.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:43 smithi132 bash[18607]: cluster 2024-04-25T20:04:42.093899+0000 mgr.x (mgr.14254) 212 : cluster [DBG] pgmap v171: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:44.545 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:04:44.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:44 smithi132 bash[18607]: audit 2024-04-25T20:04:42.928891+0000 mgr.x (mgr.14254) 213 : audit [DBG] from='client.14378 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:04:44.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:44 smithi132 bash[18607]: audit 2024-04-25T20:04:42.928891+0000 mgr.x (mgr.14254) 213 : audit [DBG] from='client.14378 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:04:45.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:45 smithi132 bash[18607]: cluster 2024-04-25T20:04:44.095739+0000 mgr.x (mgr.14254) 214 : cluster [DBG] pgmap v172: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:45.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:45 smithi132 bash[18607]: cluster 2024-04-25T20:04:44.095739+0000 mgr.x (mgr.14254) 214 : cluster [DBG] pgmap v172: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:47.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:47 smithi132 bash[18607]: cluster 2024-04-25T20:04:46.097177+0000 mgr.x (mgr.14254) 215 : cluster [DBG] pgmap v173: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:47.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:47 smithi132 bash[18607]: cluster 2024-04-25T20:04:46.097177+0000 mgr.x (mgr.14254) 215 : cluster [DBG] pgmap v173: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:47.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:47 smithi132 bash[18607]: audit 2024-04-25T20:04:46.793085+0000 mon.a (mon.0) 425 : audit [DBG] from='client.? 172.17.0.2:0/727194545' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:04:47.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:47 smithi132 bash[18607]: audit 2024-04-25T20:04:46.793085+0000 mon.a (mon.0) 425 : audit [DBG] from='client.? 172.17.0.2:0/727194545' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:04:49.301 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:04:49.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:49 smithi132 bash[18607]: cluster 2024-04-25T20:04:48.098929+0000 mgr.x (mgr.14254) 216 : cluster [DBG] pgmap v174: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:49.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:49 smithi132 bash[18607]: cluster 2024-04-25T20:04:48.098929+0000 mgr.x (mgr.14254) 216 : cluster [DBG] pgmap v174: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:51.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:51 smithi132 bash[18607]: cluster 2024-04-25T20:04:50.100337+0000 mgr.x (mgr.14254) 217 : cluster [DBG] pgmap v175: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:51.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:51 smithi132 bash[18607]: cluster 2024-04-25T20:04:50.100337+0000 mgr.x (mgr.14254) 217 : cluster [DBG] pgmap v175: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:52.005 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:04:52.005 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:04:52.708 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:04:53.709 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:04:53.721 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:53 smithi132 bash[18607]: audit 2024-04-25T20:04:52.012375+0000 mgr.x (mgr.14254) 218 : audit [DBG] from='client.14382 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:04:53.721 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:53 smithi132 bash[18607]: audit 2024-04-25T20:04:52.012375+0000 mgr.x (mgr.14254) 218 : audit [DBG] from='client.14382 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:04:53.721 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:53 smithi132 bash[18607]: cluster 2024-04-25T20:04:52.101024+0000 mgr.x (mgr.14254) 219 : cluster [DBG] pgmap v176: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:53.721 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:53 smithi132 bash[18607]: cluster 2024-04-25T20:04:52.101024+0000 mgr.x (mgr.14254) 219 : cluster [DBG] pgmap v176: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:55.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:55 smithi132 bash[18607]: cluster 2024-04-25T20:04:54.102836+0000 mgr.x (mgr.14254) 220 : cluster [DBG] pgmap v177: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:55.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:55 smithi132 bash[18607]: cluster 2024-04-25T20:04:54.102836+0000 mgr.x (mgr.14254) 220 : cluster [DBG] pgmap v177: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:57.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:57 smithi132 bash[18607]: cluster 2024-04-25T20:04:56.104321+0000 mgr.x (mgr.14254) 221 : cluster [DBG] pgmap v178: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:57.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:57 smithi132 bash[18607]: cluster 2024-04-25T20:04:56.104321+0000 mgr.x (mgr.14254) 221 : cluster [DBG] pgmap v178: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:58.464 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:04:59.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:59 smithi132 bash[18607]: cluster 2024-04-25T20:04:58.106319+0000 mgr.x (mgr.14254) 222 : cluster [DBG] pgmap v179: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:04:59.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:04:59 smithi132 bash[18607]: cluster 2024-04-25T20:04:58.106319+0000 mgr.x (mgr.14254) 222 : cluster [DBG] pgmap v179: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:01.169 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:05:01.169 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:05:01.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:01 smithi132 bash[18607]: cluster 2024-04-25T20:05:00.107081+0000 mgr.x (mgr.14254) 223 : cluster [DBG] pgmap v180: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:01.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:01 smithi132 bash[18607]: cluster 2024-04-25T20:05:00.107081+0000 mgr.x (mgr.14254) 223 : cluster [DBG] pgmap v180: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:01.810 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:05:02.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:02 smithi132 bash[18607]: audit 2024-04-25T20:05:01.176726+0000 mgr.x (mgr.14254) 224 : audit [DBG] from='client.14384 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:05:02.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:02 smithi132 bash[18607]: audit 2024-04-25T20:05:01.176726+0000 mgr.x (mgr.14254) 224 : audit [DBG] from='client.14384 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:05:02.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:02 smithi132 bash[18607]: audit 2024-04-25T20:05:02.122515+0000 mon.a (mon.0) 426 : audit [DBG] from='client.? 172.17.0.2:0/2883451169' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:05:02.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:02 smithi132 bash[18607]: audit 2024-04-25T20:05:02.122515+0000 mon.a (mon.0) 426 : audit [DBG] from='client.? 172.17.0.2:0/2883451169' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:05:02.811 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:05:03.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:03 smithi132 bash[18607]: cluster 2024-04-25T20:05:02.108442+0000 mgr.x (mgr.14254) 225 : cluster [DBG] pgmap v181: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:03.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:03 smithi132 bash[18607]: cluster 2024-04-25T20:05:02.108442+0000 mgr.x (mgr.14254) 225 : cluster [DBG] pgmap v181: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:05.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:05 smithi132 bash[18607]: cluster 2024-04-25T20:05:04.110420+0000 mgr.x (mgr.14254) 226 : cluster [DBG] pgmap v182: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:05.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:05 smithi132 bash[18607]: cluster 2024-04-25T20:05:04.110420+0000 mgr.x (mgr.14254) 226 : cluster [DBG] pgmap v182: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:07.582 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:05:07.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:07 smithi132 bash[18607]: cluster 2024-04-25T20:05:06.112018+0000 mgr.x (mgr.14254) 227 : cluster [DBG] pgmap v183: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:07.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:07 smithi132 bash[18607]: cluster 2024-04-25T20:05:06.112018+0000 mgr.x (mgr.14254) 227 : cluster [DBG] pgmap v183: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:09.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:09 smithi132 bash[18607]: cluster 2024-04-25T20:05:08.113864+0000 mgr.x (mgr.14254) 228 : cluster [DBG] pgmap v184: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:09.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:09 smithi132 bash[18607]: cluster 2024-04-25T20:05:08.113864+0000 mgr.x (mgr.14254) 228 : cluster [DBG] pgmap v184: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:10.401 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:05:10.402 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:05:11.020 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:05:11.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:11 smithi132 bash[18607]: cluster 2024-04-25T20:05:10.115254+0000 mgr.x (mgr.14254) 229 : cluster [DBG] pgmap v185: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:11.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:11 smithi132 bash[18607]: cluster 2024-04-25T20:05:10.115254+0000 mgr.x (mgr.14254) 229 : cluster [DBG] pgmap v185: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:11.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:11 smithi132 bash[18607]: audit 2024-04-25T20:05:10.406020+0000 mgr.x (mgr.14254) 230 : audit [DBG] from='client.14388 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:05:11.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:11 smithi132 bash[18607]: audit 2024-04-25T20:05:10.406020+0000 mgr.x (mgr.14254) 230 : audit [DBG] from='client.14388 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:05:12.021 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:05:13.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:13 smithi132 bash[18607]: cluster 2024-04-25T20:05:12.116708+0000 mgr.x (mgr.14254) 231 : cluster [DBG] pgmap v186: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:13.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:13 smithi132 bash[18607]: cluster 2024-04-25T20:05:12.116708+0000 mgr.x (mgr.14254) 231 : cluster [DBG] pgmap v186: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:15.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:15 smithi132 bash[18607]: cluster 2024-04-25T20:05:14.118585+0000 mgr.x (mgr.14254) 232 : cluster [DBG] pgmap v187: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:15.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:15 smithi132 bash[18607]: cluster 2024-04-25T20:05:14.118585+0000 mgr.x (mgr.14254) 232 : cluster [DBG] pgmap v187: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:16.786 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:05:17.723 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:17 smithi132 bash[18607]: cluster 2024-04-25T20:05:16.120104+0000 mgr.x (mgr.14254) 233 : cluster [DBG] pgmap v188: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:17.724 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:17 smithi132 bash[18607]: cluster 2024-04-25T20:05:16.120104+0000 mgr.x (mgr.14254) 233 : cluster [DBG] pgmap v188: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:18.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:18 smithi132 bash[18607]: audit 2024-04-25T20:05:17.455195+0000 mon.a (mon.0) 427 : audit [DBG] from='client.? 172.17.0.2:0/3684206469' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:05:18.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:18 smithi132 bash[18607]: audit 2024-04-25T20:05:17.455195+0000 mon.a (mon.0) 427 : audit [DBG] from='client.? 172.17.0.2:0/3684206469' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:05:19.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:19 smithi132 bash[18607]: cluster 2024-04-25T20:05:18.122026+0000 mgr.x (mgr.14254) 234 : cluster [DBG] pgmap v189: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:19.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:19 smithi132 bash[18607]: cluster 2024-04-25T20:05:18.122026+0000 mgr.x (mgr.14254) 234 : cluster [DBG] pgmap v189: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:19.640 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:05:19.640 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:05:20.312 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:05:21.312 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:05:21.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:21 smithi132 bash[18607]: audit 2024-04-25T20:05:19.647249+0000 mgr.x (mgr.14254) 235 : audit [DBG] from='client.14392 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:05:21.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:21 smithi132 bash[18607]: audit 2024-04-25T20:05:19.647249+0000 mgr.x (mgr.14254) 235 : audit [DBG] from='client.14392 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:05:21.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:21 smithi132 bash[18607]: cluster 2024-04-25T20:05:20.123454+0000 mgr.x (mgr.14254) 236 : cluster [DBG] pgmap v190: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:21.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:21 smithi132 bash[18607]: cluster 2024-04-25T20:05:20.123454+0000 mgr.x (mgr.14254) 236 : cluster [DBG] pgmap v190: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:23.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:23 smithi132 bash[18607]: cluster 2024-04-25T20:05:22.124896+0000 mgr.x (mgr.14254) 237 : cluster [DBG] pgmap v191: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:23.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:23 smithi132 bash[18607]: cluster 2024-04-25T20:05:22.124896+0000 mgr.x (mgr.14254) 237 : cluster [DBG] pgmap v191: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:25.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:25 smithi132 bash[18607]: cluster 2024-04-25T20:05:24.125849+0000 mgr.x (mgr.14254) 238 : cluster [DBG] pgmap v192: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:25.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:25 smithi132 bash[18607]: cluster 2024-04-25T20:05:24.125849+0000 mgr.x (mgr.14254) 238 : cluster [DBG] pgmap v192: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:26.068 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:05:27.504 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:27 smithi132 bash[18607]: cluster 2024-04-25T20:05:26.126622+0000 mgr.x (mgr.14254) 239 : cluster [DBG] pgmap v193: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:27.504 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:27 smithi132 bash[18607]: cluster 2024-04-25T20:05:26.126622+0000 mgr.x (mgr.14254) 239 : cluster [DBG] pgmap v193: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:28.913 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:05:28.913 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:05:29.525 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:05:29.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:29 smithi132 bash[18607]: cluster 2024-04-25T20:05:28.128559+0000 mgr.x (mgr.14254) 240 : cluster [DBG] pgmap v194: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:29.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:29 smithi132 bash[18607]: cluster 2024-04-25T20:05:28.128559+0000 mgr.x (mgr.14254) 240 : cluster [DBG] pgmap v194: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:30.526 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:05:30.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:30 smithi132 bash[18607]: audit 2024-04-25T20:05:28.917886+0000 mgr.x (mgr.14254) 241 : audit [DBG] from='client.14394 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:05:30.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:30 smithi132 bash[18607]: audit 2024-04-25T20:05:28.917886+0000 mgr.x (mgr.14254) 241 : audit [DBG] from='client.14394 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:05:31.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:31 smithi132 bash[18607]: cluster 2024-04-25T20:05:30.130185+0000 mgr.x (mgr.14254) 242 : cluster [DBG] pgmap v195: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:31.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:31 smithi132 bash[18607]: cluster 2024-04-25T20:05:30.130185+0000 mgr.x (mgr.14254) 242 : cluster [DBG] pgmap v195: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:33.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:33 smithi132 bash[18607]: cluster 2024-04-25T20:05:32.131742+0000 mgr.x (mgr.14254) 243 : cluster [DBG] pgmap v196: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:33.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:33 smithi132 bash[18607]: cluster 2024-04-25T20:05:32.131742+0000 mgr.x (mgr.14254) 243 : cluster [DBG] pgmap v196: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:33.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:33 smithi132 bash[18607]: audit 2024-04-25T20:05:32.787181+0000 mon.a (mon.0) 428 : audit [DBG] from='client.? 172.17.0.2:0/66993697' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:05:33.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:33 smithi132 bash[18607]: audit 2024-04-25T20:05:32.787181+0000 mon.a (mon.0) 428 : audit [DBG] from='client.? 172.17.0.2:0/66993697' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:05:35.291 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:05:35.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:35 smithi132 bash[18607]: cluster 2024-04-25T20:05:34.133791+0000 mgr.x (mgr.14254) 244 : cluster [DBG] pgmap v197: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:35.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:35 smithi132 bash[18607]: cluster 2024-04-25T20:05:34.133791+0000 mgr.x (mgr.14254) 244 : cluster [DBG] pgmap v197: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:37.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:37 smithi132 bash[18607]: cluster 2024-04-25T20:05:36.135386+0000 mgr.x (mgr.14254) 245 : cluster [DBG] pgmap v198: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:37.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:37 smithi132 bash[18607]: cluster 2024-04-25T20:05:36.135386+0000 mgr.x (mgr.14254) 245 : cluster [DBG] pgmap v198: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:38.070 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:05:38.070 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:05:38.754 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:05:39.755 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:05:39.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:39 smithi132 bash[18607]: audit 2024-04-25T20:05:38.074955+0000 mgr.x (mgr.14254) 246 : audit [DBG] from='client.14398 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:05:39.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:39 smithi132 bash[18607]: audit 2024-04-25T20:05:38.074955+0000 mgr.x (mgr.14254) 246 : audit [DBG] from='client.14398 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:05:39.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:39 smithi132 bash[18607]: cluster 2024-04-25T20:05:38.136450+0000 mgr.x (mgr.14254) 247 : cluster [DBG] pgmap v199: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:39.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:39 smithi132 bash[18607]: cluster 2024-04-25T20:05:38.136450+0000 mgr.x (mgr.14254) 247 : cluster [DBG] pgmap v199: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:41.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:41 smithi132 bash[18607]: cluster 2024-04-25T20:05:40.137820+0000 mgr.x (mgr.14254) 248 : cluster [DBG] pgmap v200: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:41.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:41 smithi132 bash[18607]: cluster 2024-04-25T20:05:40.137820+0000 mgr.x (mgr.14254) 248 : cluster [DBG] pgmap v200: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:43.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:43 smithi132 bash[18607]: cluster 2024-04-25T20:05:42.139232+0000 mgr.x (mgr.14254) 249 : cluster [DBG] pgmap v201: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:43.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:43 smithi132 bash[18607]: cluster 2024-04-25T20:05:42.139232+0000 mgr.x (mgr.14254) 249 : cluster [DBG] pgmap v201: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:44.522 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:05:45.726 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:45 smithi132 bash[18607]: cluster 2024-04-25T20:05:44.141131+0000 mgr.x (mgr.14254) 250 : cluster [DBG] pgmap v202: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:45.726 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:45 smithi132 bash[18607]: cluster 2024-04-25T20:05:44.141131+0000 mgr.x (mgr.14254) 250 : cluster [DBG] pgmap v202: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:47.326 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:05:47.326 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:05:47.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:47 smithi132 bash[18607]: cluster 2024-04-25T20:05:46.142642+0000 mgr.x (mgr.14254) 251 : cluster [DBG] pgmap v203: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:47.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:47 smithi132 bash[18607]: cluster 2024-04-25T20:05:46.142642+0000 mgr.x (mgr.14254) 251 : cluster [DBG] pgmap v203: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:48.024 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:05:48.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:48 smithi132 bash[18607]: audit 2024-04-25T20:05:47.330732+0000 mgr.x (mgr.14254) 252 : audit [DBG] from='client.14400 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:05:48.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:48 smithi132 bash[18607]: audit 2024-04-25T20:05:47.330732+0000 mgr.x (mgr.14254) 252 : audit [DBG] from='client.14400 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:05:48.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:48 smithi132 bash[18607]: audit 2024-04-25T20:05:48.121865+0000 mon.a (mon.0) 429 : audit [DBG] from='client.? 172.17.0.2:0/4242739592' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:05:48.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:48 smithi132 bash[18607]: audit 2024-04-25T20:05:48.121865+0000 mon.a (mon.0) 429 : audit [DBG] from='client.? 172.17.0.2:0/4242739592' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:05:49.025 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:05:49.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:49 smithi132 bash[18607]: cluster 2024-04-25T20:05:48.143548+0000 mgr.x (mgr.14254) 253 : cluster [DBG] pgmap v204: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:49.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:49 smithi132 bash[18607]: cluster 2024-04-25T20:05:48.143548+0000 mgr.x (mgr.14254) 253 : cluster [DBG] pgmap v204: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:51.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:51 smithi132 bash[18607]: cluster 2024-04-25T20:05:50.144931+0000 mgr.x (mgr.14254) 254 : cluster [DBG] pgmap v205: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:51.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:51 smithi132 bash[18607]: cluster 2024-04-25T20:05:50.144931+0000 mgr.x (mgr.14254) 254 : cluster [DBG] pgmap v205: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:53.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:53 smithi132 bash[18607]: cluster 2024-04-25T20:05:52.146343+0000 mgr.x (mgr.14254) 255 : cluster [DBG] pgmap v206: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:53.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:53 smithi132 bash[18607]: cluster 2024-04-25T20:05:52.146343+0000 mgr.x (mgr.14254) 255 : cluster [DBG] pgmap v206: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:53.784 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:05:55.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:55 smithi132 bash[18607]: cluster 2024-04-25T20:05:54.147490+0000 mgr.x (mgr.14254) 256 : cluster [DBG] pgmap v207: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:55.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:55 smithi132 bash[18607]: cluster 2024-04-25T20:05:54.147490+0000 mgr.x (mgr.14254) 256 : cluster [DBG] pgmap v207: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:56.493 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:05:56.493 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:05:56.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:56 smithi132 bash[18607]: cluster 2024-04-25T20:05:56.148465+0000 mgr.x (mgr.14254) 257 : cluster [DBG] pgmap v208: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:56.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:56 smithi132 bash[18607]: cluster 2024-04-25T20:05:56.148465+0000 mgr.x (mgr.14254) 257 : cluster [DBG] pgmap v208: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:57.157 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:05:58.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:57 smithi132 bash[18607]: audit 2024-04-25T20:05:56.500197+0000 mgr.x (mgr.14254) 258 : audit [DBG] from='client.14404 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:05:58.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:57 smithi132 bash[18607]: audit 2024-04-25T20:05:56.500197+0000 mgr.x (mgr.14254) 258 : audit [DBG] from='client.14404 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:05:58.158 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:05:58.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:58 smithi132 bash[18607]: cluster 2024-04-25T20:05:58.150379+0000 mgr.x (mgr.14254) 259 : cluster [DBG] pgmap v209: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:05:58.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:05:58 smithi132 bash[18607]: cluster 2024-04-25T20:05:58.150379+0000 mgr.x (mgr.14254) 259 : cluster [DBG] pgmap v209: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:01.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:01 smithi132 bash[18607]: cluster 2024-04-25T20:06:00.151940+0000 mgr.x (mgr.14254) 260 : cluster [DBG] pgmap v210: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:01.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:01 smithi132 bash[18607]: cluster 2024-04-25T20:06:00.151940+0000 mgr.x (mgr.14254) 260 : cluster [DBG] pgmap v210: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:02.916 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:06:03.491 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:03 smithi132 bash[18607]: cluster 2024-04-25T20:06:02.153502+0000 mgr.x (mgr.14254) 261 : cluster [DBG] pgmap v211: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:03.491 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:03 smithi132 bash[18607]: cluster 2024-04-25T20:06:02.153502+0000 mgr.x (mgr.14254) 261 : cluster [DBG] pgmap v211: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:04.514 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:04 smithi132 bash[18607]: audit 2024-04-25T20:06:03.456594+0000 mon.a (mon.0) 430 : audit [DBG] from='client.? 172.17.0.2:0/2863840644' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:06:04.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:04 smithi132 bash[18607]: audit 2024-04-25T20:06:03.456594+0000 mon.a (mon.0) 430 : audit [DBG] from='client.? 172.17.0.2:0/2863840644' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:06:05.472 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:05 smithi132 bash[18607]: cluster 2024-04-25T20:06:04.155497+0000 mgr.x (mgr.14254) 262 : cluster [DBG] pgmap v212: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:05.472 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:05 smithi132 bash[18607]: cluster 2024-04-25T20:06:04.155497+0000 mgr.x (mgr.14254) 262 : cluster [DBG] pgmap v212: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:05.861 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:06:05.862 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:06:06.542 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:06:07.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:07 smithi132 bash[18607]: audit 2024-04-25T20:06:05.868792+0000 mgr.x (mgr.14254) 263 : audit [DBG] from='client.14408 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:06:07.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:07 smithi132 bash[18607]: audit 2024-04-25T20:06:05.868792+0000 mgr.x (mgr.14254) 263 : audit [DBG] from='client.14408 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:06:07.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:07 smithi132 bash[18607]: cluster 2024-04-25T20:06:06.156937+0000 mgr.x (mgr.14254) 264 : cluster [DBG] pgmap v213: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:07.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:07 smithi132 bash[18607]: cluster 2024-04-25T20:06:06.156937+0000 mgr.x (mgr.14254) 264 : cluster [DBG] pgmap v213: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:07.543 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:06:09.514 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:09 smithi132 bash[18607]: cluster 2024-04-25T20:06:08.158667+0000 mgr.x (mgr.14254) 265 : cluster [DBG] pgmap v214: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:09.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:09 smithi132 bash[18607]: cluster 2024-04-25T20:06:08.158667+0000 mgr.x (mgr.14254) 265 : cluster [DBG] pgmap v214: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:11.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:11 smithi132 bash[18607]: cluster 2024-04-25T20:06:10.160029+0000 mgr.x (mgr.14254) 266 : cluster [DBG] pgmap v215: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:11.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:11 smithi132 bash[18607]: cluster 2024-04-25T20:06:10.160029+0000 mgr.x (mgr.14254) 266 : cluster [DBG] pgmap v215: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:12.294 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:06:13.438 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:13 smithi132 bash[18607]: cluster 2024-04-25T20:06:12.161440+0000 mgr.x (mgr.14254) 267 : cluster [DBG] pgmap v216: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:13.438 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:13 smithi132 bash[18607]: cluster 2024-04-25T20:06:12.161440+0000 mgr.x (mgr.14254) 267 : cluster [DBG] pgmap v216: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:15.046 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:06:15.046 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:06:15.514 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:15 smithi132 bash[18607]: cluster 2024-04-25T20:06:14.163343+0000 mgr.x (mgr.14254) 268 : cluster [DBG] pgmap v217: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:15.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:15 smithi132 bash[18607]: cluster 2024-04-25T20:06:14.163343+0000 mgr.x (mgr.14254) 268 : cluster [DBG] pgmap v217: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:15.729 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:06:16.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:16 smithi132 bash[18607]: audit 2024-04-25T20:06:15.051200+0000 mgr.x (mgr.14254) 269 : audit [DBG] from='client.14410 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:06:16.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:16 smithi132 bash[18607]: audit 2024-04-25T20:06:15.051200+0000 mgr.x (mgr.14254) 269 : audit [DBG] from='client.14410 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:06:16.729 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:06:17.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:17 smithi132 bash[18607]: cluster 2024-04-25T20:06:16.164898+0000 mgr.x (mgr.14254) 270 : cluster [DBG] pgmap v218: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:17.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:17 smithi132 bash[18607]: cluster 2024-04-25T20:06:16.164898+0000 mgr.x (mgr.14254) 270 : cluster [DBG] pgmap v218: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:19.514 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:19 smithi132 bash[18607]: cluster 2024-04-25T20:06:18.166775+0000 mgr.x (mgr.14254) 271 : cluster [DBG] pgmap v219: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:19.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:19 smithi132 bash[18607]: cluster 2024-04-25T20:06:18.166775+0000 mgr.x (mgr.14254) 271 : cluster [DBG] pgmap v219: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:19.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:19 smithi132 bash[18607]: audit 2024-04-25T20:06:18.790998+0000 mon.a (mon.0) 431 : audit [DBG] from='client.? 172.17.0.2:0/4019779965' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:06:19.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:19 smithi132 bash[18607]: audit 2024-04-25T20:06:18.790998+0000 mon.a (mon.0) 431 : audit [DBG] from='client.? 172.17.0.2:0/4019779965' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:06:21.490 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:06:21.513 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:21 smithi132 bash[18607]: cluster 2024-04-25T20:06:20.168238+0000 mgr.x (mgr.14254) 272 : cluster [DBG] pgmap v220: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:21.513 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:21 smithi132 bash[18607]: cluster 2024-04-25T20:06:20.168238+0000 mgr.x (mgr.14254) 272 : cluster [DBG] pgmap v220: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:23.514 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:23 smithi132 bash[18607]: cluster 2024-04-25T20:06:22.169779+0000 mgr.x (mgr.14254) 273 : cluster [DBG] pgmap v221: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:23.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:23 smithi132 bash[18607]: cluster 2024-04-25T20:06:22.169779+0000 mgr.x (mgr.14254) 273 : cluster [DBG] pgmap v221: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:24.232 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:06:24.232 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:06:24.842 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:06:25.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:25 smithi132 bash[18607]: cluster 2024-04-25T20:06:24.171711+0000 mgr.x (mgr.14254) 274 : cluster [DBG] pgmap v222: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:25.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:25 smithi132 bash[18607]: cluster 2024-04-25T20:06:24.171711+0000 mgr.x (mgr.14254) 274 : cluster [DBG] pgmap v222: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:25.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:25 smithi132 bash[18607]: audit 2024-04-25T20:06:24.236874+0000 mgr.x (mgr.14254) 275 : audit [DBG] from='client.14414 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:06:25.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:25 smithi132 bash[18607]: audit 2024-04-25T20:06:24.236874+0000 mgr.x (mgr.14254) 275 : audit [DBG] from='client.14414 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:06:25.843 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:06:27.514 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:27 smithi132 bash[18607]: cluster 2024-04-25T20:06:26.173329+0000 mgr.x (mgr.14254) 276 : cluster [DBG] pgmap v223: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:27.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:27 smithi132 bash[18607]: cluster 2024-04-25T20:06:26.173329+0000 mgr.x (mgr.14254) 276 : cluster [DBG] pgmap v223: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:29.514 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:29 smithi132 bash[18607]: cluster 2024-04-25T20:06:28.175252+0000 mgr.x (mgr.14254) 277 : cluster [DBG] pgmap v224: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:29.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:29 smithi132 bash[18607]: cluster 2024-04-25T20:06:28.175252+0000 mgr.x (mgr.14254) 277 : cluster [DBG] pgmap v224: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:30.605 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:06:31.503 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:31 smithi132 bash[18607]: cluster 2024-04-25T20:06:30.176749+0000 mgr.x (mgr.14254) 278 : cluster [DBG] pgmap v225: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:31.503 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:31 smithi132 bash[18607]: cluster 2024-04-25T20:06:30.176749+0000 mgr.x (mgr.14254) 278 : cluster [DBG] pgmap v225: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:33.331 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:06:33.332 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:06:33.514 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:33 smithi132 bash[18607]: cluster 2024-04-25T20:06:32.178115+0000 mgr.x (mgr.14254) 279 : cluster [DBG] pgmap v226: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:33.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:33 smithi132 bash[18607]: cluster 2024-04-25T20:06:32.178115+0000 mgr.x (mgr.14254) 279 : cluster [DBG] pgmap v226: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:33.973 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:06:34.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:34 smithi132 bash[18607]: audit 2024-04-25T20:06:33.337644+0000 mgr.x (mgr.14254) 280 : audit [DBG] from='client.14416 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:06:34.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:34 smithi132 bash[18607]: audit 2024-04-25T20:06:33.337644+0000 mgr.x (mgr.14254) 280 : audit [DBG] from='client.14416 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:06:34.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:34 smithi132 bash[18607]: audit 2024-04-25T20:06:34.118621+0000 mon.a (mon.0) 432 : audit [DBG] from='client.? 172.17.0.2:0/2297746752' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:06:34.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:34 smithi132 bash[18607]: audit 2024-04-25T20:06:34.118621+0000 mon.a (mon.0) 432 : audit [DBG] from='client.? 172.17.0.2:0/2297746752' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:06:34.974 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:06:35.514 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:35 smithi132 bash[18607]: cluster 2024-04-25T20:06:34.179953+0000 mgr.x (mgr.14254) 281 : cluster [DBG] pgmap v227: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:35.514 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:35 smithi132 bash[18607]: cluster 2024-04-25T20:06:34.179953+0000 mgr.x (mgr.14254) 281 : cluster [DBG] pgmap v227: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:37.514 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:37 smithi132 bash[18607]: cluster 2024-04-25T20:06:36.181492+0000 mgr.x (mgr.14254) 282 : cluster [DBG] pgmap v228: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:37.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:37 smithi132 bash[18607]: cluster 2024-04-25T20:06:36.181492+0000 mgr.x (mgr.14254) 282 : cluster [DBG] pgmap v228: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:39.514 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:39 smithi132 bash[18607]: cluster 2024-04-25T20:06:38.183474+0000 mgr.x (mgr.14254) 283 : cluster [DBG] pgmap v229: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:39.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:39 smithi132 bash[18607]: cluster 2024-04-25T20:06:38.183474+0000 mgr.x (mgr.14254) 283 : cluster [DBG] pgmap v229: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:39.730 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:06:41.514 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:41 smithi132 bash[18607]: cluster 2024-04-25T20:06:40.184909+0000 mgr.x (mgr.14254) 284 : cluster [DBG] pgmap v230: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:41.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:41 smithi132 bash[18607]: cluster 2024-04-25T20:06:40.184909+0000 mgr.x (mgr.14254) 284 : cluster [DBG] pgmap v230: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:42.539 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:06:42.539 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:06:43.206 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:06:43.514 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:43 smithi132 bash[18607]: cluster 2024-04-25T20:06:42.185806+0000 mgr.x (mgr.14254) 285 : cluster [DBG] pgmap v231: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:43.514 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:43 smithi132 bash[18607]: cluster 2024-04-25T20:06:42.185806+0000 mgr.x (mgr.14254) 285 : cluster [DBG] pgmap v231: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:44.207 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:06:44.417 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:44 smithi132 bash[18607]: audit 2024-04-25T20:06:42.544783+0000 mgr.x (mgr.14254) 286 : audit [DBG] from='client.14420 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:06:44.417 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:44 smithi132 bash[18607]: audit 2024-04-25T20:06:42.544783+0000 mgr.x (mgr.14254) 286 : audit [DBG] from='client.14420 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:06:45.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:45 smithi132 bash[18607]: cluster 2024-04-25T20:06:44.187531+0000 mgr.x (mgr.14254) 287 : cluster [DBG] pgmap v232: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:45.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:45 smithi132 bash[18607]: cluster 2024-04-25T20:06:44.187531+0000 mgr.x (mgr.14254) 287 : cluster [DBG] pgmap v232: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:47.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:47 smithi132 bash[18607]: cluster 2024-04-25T20:06:46.188916+0000 mgr.x (mgr.14254) 288 : cluster [DBG] pgmap v233: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:47.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:47 smithi132 bash[18607]: cluster 2024-04-25T20:06:46.188916+0000 mgr.x (mgr.14254) 288 : cluster [DBG] pgmap v233: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:48.966 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:06:49.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:49 smithi132 bash[18607]: cluster 2024-04-25T20:06:48.190621+0000 mgr.x (mgr.14254) 289 : cluster [DBG] pgmap v234: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:49.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:49 smithi132 bash[18607]: cluster 2024-04-25T20:06:48.190621+0000 mgr.x (mgr.14254) 289 : cluster [DBG] pgmap v234: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:50.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:50 smithi132 bash[18607]: audit 2024-04-25T20:06:49.453682+0000 mon.a (mon.0) 433 : audit [DBG] from='client.? 172.17.0.2:0/301729617' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:06:50.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:50 smithi132 bash[18607]: audit 2024-04-25T20:06:49.453682+0000 mon.a (mon.0) 433 : audit [DBG] from='client.? 172.17.0.2:0/301729617' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:06:51.551 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:51 smithi132 bash[18607]: cluster 2024-04-25T20:06:50.191375+0000 mgr.x (mgr.14254) 290 : cluster [DBG] pgmap v235: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:51.551 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:51 smithi132 bash[18607]: cluster 2024-04-25T20:06:50.191375+0000 mgr.x (mgr.14254) 290 : cluster [DBG] pgmap v235: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:51.914 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:06:51.914 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:06:52.561 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:06:53.562 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:06:53.574 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:53 smithi132 bash[18607]: audit 2024-04-25T20:06:51.919126+0000 mgr.x (mgr.14254) 291 : audit [DBG] from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:06:53.574 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:53 smithi132 bash[18607]: audit 2024-04-25T20:06:51.919126+0000 mgr.x (mgr.14254) 291 : audit [DBG] from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:06:53.574 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:53 smithi132 bash[18607]: cluster 2024-04-25T20:06:52.192780+0000 mgr.x (mgr.14254) 292 : cluster [DBG] pgmap v236: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:53.574 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:53 smithi132 bash[18607]: cluster 2024-04-25T20:06:52.192780+0000 mgr.x (mgr.14254) 292 : cluster [DBG] pgmap v236: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:55.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:55 smithi132 bash[18607]: cluster 2024-04-25T20:06:54.194578+0000 mgr.x (mgr.14254) 293 : cluster [DBG] pgmap v237: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:55.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:55 smithi132 bash[18607]: cluster 2024-04-25T20:06:54.194578+0000 mgr.x (mgr.14254) 293 : cluster [DBG] pgmap v237: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:57.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:57 smithi132 bash[18607]: cluster 2024-04-25T20:06:56.195983+0000 mgr.x (mgr.14254) 294 : cluster [DBG] pgmap v238: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:57.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:57 smithi132 bash[18607]: cluster 2024-04-25T20:06:56.195983+0000 mgr.x (mgr.14254) 294 : cluster [DBG] pgmap v238: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:58.320 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:06:59.497 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:59 smithi132 bash[18607]: cluster 2024-04-25T20:06:58.197823+0000 mgr.x (mgr.14254) 295 : cluster [DBG] pgmap v239: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:06:59.497 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:06:59 smithi132 bash[18607]: cluster 2024-04-25T20:06:58.197823+0000 mgr.x (mgr.14254) 295 : cluster [DBG] pgmap v239: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:01.280 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:07:01.280 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:07:01.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:01 smithi132 bash[18607]: cluster 2024-04-25T20:07:00.199225+0000 mgr.x (mgr.14254) 296 : cluster [DBG] pgmap v240: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:01.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:01 smithi132 bash[18607]: cluster 2024-04-25T20:07:00.199225+0000 mgr.x (mgr.14254) 296 : cluster [DBG] pgmap v240: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:01.994 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:07:02.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:02 smithi132 bash[18607]: audit 2024-04-25T20:07:01.285543+0000 mgr.x (mgr.14254) 297 : audit [DBG] from='client.14426 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:07:02.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:02 smithi132 bash[18607]: audit 2024-04-25T20:07:01.285543+0000 mgr.x (mgr.14254) 297 : audit [DBG] from='client.14426 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:07:02.995 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:07:03.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:03 smithi132 bash[18607]: cluster 2024-04-25T20:07:02.200635+0000 mgr.x (mgr.14254) 298 : cluster [DBG] pgmap v241: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:03.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:03 smithi132 bash[18607]: cluster 2024-04-25T20:07:02.200635+0000 mgr.x (mgr.14254) 298 : cluster [DBG] pgmap v241: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:05.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:05 smithi132 bash[18607]: cluster 2024-04-25T20:07:04.202520+0000 mgr.x (mgr.14254) 299 : cluster [DBG] pgmap v242: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:05.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:05 smithi132 bash[18607]: cluster 2024-04-25T20:07:04.202520+0000 mgr.x (mgr.14254) 299 : cluster [DBG] pgmap v242: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:05.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:05 smithi132 bash[18607]: audit 2024-04-25T20:07:04.786158+0000 mon.a (mon.0) 434 : audit [DBG] from='client.? 172.17.0.2:0/1251847505' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:07:05.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:05 smithi132 bash[18607]: audit 2024-04-25T20:07:04.786158+0000 mon.a (mon.0) 434 : audit [DBG] from='client.? 172.17.0.2:0/1251847505' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:07:07.750 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:07:07.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:07 smithi132 bash[18607]: cluster 2024-04-25T20:07:06.203999+0000 mgr.x (mgr.14254) 300 : cluster [DBG] pgmap v243: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:07.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:07 smithi132 bash[18607]: cluster 2024-04-25T20:07:06.203999+0000 mgr.x (mgr.14254) 300 : cluster [DBG] pgmap v243: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:09.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:09 smithi132 bash[18607]: cluster 2024-04-25T20:07:08.205788+0000 mgr.x (mgr.14254) 301 : cluster [DBG] pgmap v244: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:09.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:09 smithi132 bash[18607]: cluster 2024-04-25T20:07:08.205788+0000 mgr.x (mgr.14254) 301 : cluster [DBG] pgmap v244: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:10.409 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:07:10.410 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:07:11.072 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:07:11.514 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:11 smithi132 bash[18607]: cluster 2024-04-25T20:07:10.206497+0000 mgr.x (mgr.14254) 302 : cluster [DBG] pgmap v245: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:11.514 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:11 smithi132 bash[18607]: cluster 2024-04-25T20:07:10.206497+0000 mgr.x (mgr.14254) 302 : cluster [DBG] pgmap v245: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:11.514 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:11 smithi132 bash[18607]: audit 2024-04-25T20:07:10.416946+0000 mgr.x (mgr.14254) 303 : audit [DBG] from='client.14430 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:07:11.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:11 smithi132 bash[18607]: audit 2024-04-25T20:07:10.416946+0000 mgr.x (mgr.14254) 303 : audit [DBG] from='client.14430 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:07:12.073 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:07:13.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:13 smithi132 bash[18607]: cluster 2024-04-25T20:07:12.207808+0000 mgr.x (mgr.14254) 304 : cluster [DBG] pgmap v246: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:13.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:13 smithi132 bash[18607]: cluster 2024-04-25T20:07:12.207808+0000 mgr.x (mgr.14254) 304 : cluster [DBG] pgmap v246: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:15.514 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:15 smithi132 bash[18607]: cluster 2024-04-25T20:07:14.209615+0000 mgr.x (mgr.14254) 305 : cluster [DBG] pgmap v247: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:15.514 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:15 smithi132 bash[18607]: cluster 2024-04-25T20:07:14.209615+0000 mgr.x (mgr.14254) 305 : cluster [DBG] pgmap v247: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:16.833 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:07:17.686 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:17 smithi132 bash[18607]: cluster 2024-04-25T20:07:16.211067+0000 mgr.x (mgr.14254) 306 : cluster [DBG] pgmap v248: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:17.687 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:17 smithi132 bash[18607]: cluster 2024-04-25T20:07:16.211067+0000 mgr.x (mgr.14254) 306 : cluster [DBG] pgmap v248: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:19.493 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:07:19.493 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:07:19.506 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:19 smithi132 bash[18607]: cluster 2024-04-25T20:07:18.212951+0000 mgr.x (mgr.14254) 307 : cluster [DBG] pgmap v249: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:19.506 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:19 smithi132 bash[18607]: cluster 2024-04-25T20:07:18.212951+0000 mgr.x (mgr.14254) 307 : cluster [DBG] pgmap v249: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:20.152 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:07:20.514 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:20 smithi132 bash[18607]: audit 2024-04-25T20:07:19.500887+0000 mgr.x (mgr.14254) 308 : audit [DBG] from='client.14432 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:07:20.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:20 smithi132 bash[18607]: audit 2024-04-25T20:07:19.500887+0000 mgr.x (mgr.14254) 308 : audit [DBG] from='client.14432 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:07:20.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:20 smithi132 bash[18607]: audit 2024-04-25T20:07:20.120131+0000 mon.a (mon.0) 435 : audit [DBG] from='client.? 172.17.0.2:0/547996154' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:07:20.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:20 smithi132 bash[18607]: audit 2024-04-25T20:07:20.120131+0000 mon.a (mon.0) 435 : audit [DBG] from='client.? 172.17.0.2:0/547996154' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:07:21.153 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:07:21.514 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:21 smithi132 bash[18607]: cluster 2024-04-25T20:07:20.213933+0000 mgr.x (mgr.14254) 309 : cluster [DBG] pgmap v250: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:21.514 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:21 smithi132 bash[18607]: cluster 2024-04-25T20:07:20.213933+0000 mgr.x (mgr.14254) 309 : cluster [DBG] pgmap v250: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:23.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:23 smithi132 bash[18607]: cluster 2024-04-25T20:07:22.215288+0000 mgr.x (mgr.14254) 310 : cluster [DBG] pgmap v251: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:23.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:23 smithi132 bash[18607]: cluster 2024-04-25T20:07:22.215288+0000 mgr.x (mgr.14254) 310 : cluster [DBG] pgmap v251: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:25.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:25 smithi132 bash[18607]: cluster 2024-04-25T20:07:24.217313+0000 mgr.x (mgr.14254) 311 : cluster [DBG] pgmap v252: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:25.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:25 smithi132 bash[18607]: cluster 2024-04-25T20:07:24.217313+0000 mgr.x (mgr.14254) 311 : cluster [DBG] pgmap v252: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:25.907 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:07:27.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:27 smithi132 bash[18607]: cluster 2024-04-25T20:07:26.218894+0000 mgr.x (mgr.14254) 312 : cluster [DBG] pgmap v253: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:27.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:27 smithi132 bash[18607]: cluster 2024-04-25T20:07:26.218894+0000 mgr.x (mgr.14254) 312 : cluster [DBG] pgmap v253: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:28.656 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:07:28.656 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:07:29.322 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:07:29.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:29 smithi132 bash[18607]: cluster 2024-04-25T20:07:28.220983+0000 mgr.x (mgr.14254) 313 : cluster [DBG] pgmap v254: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:29.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:29 smithi132 bash[18607]: cluster 2024-04-25T20:07:28.220983+0000 mgr.x (mgr.14254) 313 : cluster [DBG] pgmap v254: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:30.323 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:07:30.422 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:30 smithi132 bash[18607]: audit 2024-04-25T20:07:28.663713+0000 mgr.x (mgr.14254) 314 : audit [DBG] from='client.14436 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:07:30.422 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:30 smithi132 bash[18607]: audit 2024-04-25T20:07:28.663713+0000 mgr.x (mgr.14254) 314 : audit [DBG] from='client.14436 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:07:31.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:31 smithi132 bash[18607]: cluster 2024-04-25T20:07:30.222351+0000 mgr.x (mgr.14254) 315 : cluster [DBG] pgmap v255: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:31.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:31 smithi132 bash[18607]: cluster 2024-04-25T20:07:30.222351+0000 mgr.x (mgr.14254) 315 : cluster [DBG] pgmap v255: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:33.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:33 smithi132 bash[18607]: cluster 2024-04-25T20:07:32.223720+0000 mgr.x (mgr.14254) 316 : cluster [DBG] pgmap v256: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:33.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:33 smithi132 bash[18607]: cluster 2024-04-25T20:07:32.223720+0000 mgr.x (mgr.14254) 316 : cluster [DBG] pgmap v256: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:35.082 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:07:35.489 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:35 smithi132 bash[18607]: cluster 2024-04-25T20:07:34.225637+0000 mgr.x (mgr.14254) 317 : cluster [DBG] pgmap v257: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:35.489 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:35 smithi132 bash[18607]: cluster 2024-04-25T20:07:34.225637+0000 mgr.x (mgr.14254) 317 : cluster [DBG] pgmap v257: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:36.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:36 smithi132 bash[18607]: audit 2024-04-25T20:07:35.455313+0000 mon.a (mon.0) 436 : audit [DBG] from='client.? 172.17.0.2:0/3601346785' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:07:36.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:36 smithi132 bash[18607]: audit 2024-04-25T20:07:35.455313+0000 mon.a (mon.0) 436 : audit [DBG] from='client.? 172.17.0.2:0/3601346785' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:07:37.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:37 smithi132 bash[18607]: cluster 2024-04-25T20:07:36.227024+0000 mgr.x (mgr.14254) 318 : cluster [DBG] pgmap v258: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:37.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:37 smithi132 bash[18607]: cluster 2024-04-25T20:07:36.227024+0000 mgr.x (mgr.14254) 318 : cluster [DBG] pgmap v258: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:37.932 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:07:37.932 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:07:38.571 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:07:39.572 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:07:39.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:39 smithi132 bash[18607]: audit 2024-04-25T20:07:37.940548+0000 mgr.x (mgr.14254) 319 : audit [DBG] from='client.14440 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:07:39.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:39 smithi132 bash[18607]: audit 2024-04-25T20:07:37.940548+0000 mgr.x (mgr.14254) 319 : audit [DBG] from='client.14440 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:07:39.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:39 smithi132 bash[18607]: cluster 2024-04-25T20:07:38.228911+0000 mgr.x (mgr.14254) 320 : cluster [DBG] pgmap v259: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:39.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:39 smithi132 bash[18607]: cluster 2024-04-25T20:07:38.228911+0000 mgr.x (mgr.14254) 320 : cluster [DBG] pgmap v259: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:41.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:41 smithi132 bash[18607]: cluster 2024-04-25T20:07:40.230303+0000 mgr.x (mgr.14254) 321 : cluster [DBG] pgmap v260: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:41.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:41 smithi132 bash[18607]: cluster 2024-04-25T20:07:40.230303+0000 mgr.x (mgr.14254) 321 : cluster [DBG] pgmap v260: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:43.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:43 smithi132 bash[18607]: cluster 2024-04-25T20:07:42.231693+0000 mgr.x (mgr.14254) 322 : cluster [DBG] pgmap v261: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:43.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:43 smithi132 bash[18607]: cluster 2024-04-25T20:07:42.231693+0000 mgr.x (mgr.14254) 322 : cluster [DBG] pgmap v261: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:44.333 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:07:45.423 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:45 smithi132 bash[18607]: cluster 2024-04-25T20:07:44.233576+0000 mgr.x (mgr.14254) 323 : cluster [DBG] pgmap v262: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:45.423 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:45 smithi132 bash[18607]: cluster 2024-04-25T20:07:44.233576+0000 mgr.x (mgr.14254) 323 : cluster [DBG] pgmap v262: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:47.121 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:07:47.121 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:07:47.514 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:47 smithi132 bash[18607]: cluster 2024-04-25T20:07:46.234998+0000 mgr.x (mgr.14254) 324 : cluster [DBG] pgmap v263: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:47.514 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:47 smithi132 bash[18607]: cluster 2024-04-25T20:07:46.234998+0000 mgr.x (mgr.14254) 324 : cluster [DBG] pgmap v263: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:47.745 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:07:48.747 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:07:48.758 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:48 smithi132 bash[18607]: audit 2024-04-25T20:07:47.126544+0000 mgr.x (mgr.14254) 325 : audit [DBG] from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:07:48.758 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:48 smithi132 bash[18607]: audit 2024-04-25T20:07:47.126544+0000 mgr.x (mgr.14254) 325 : audit [DBG] from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:07:49.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:49 smithi132 bash[18607]: cluster 2024-04-25T20:07:48.236833+0000 mgr.x (mgr.14254) 326 : cluster [DBG] pgmap v264: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:49.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:49 smithi132 bash[18607]: cluster 2024-04-25T20:07:48.236833+0000 mgr.x (mgr.14254) 326 : cluster [DBG] pgmap v264: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:51.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:51 smithi132 bash[18607]: cluster 2024-04-25T20:07:50.238290+0000 mgr.x (mgr.14254) 327 : cluster [DBG] pgmap v265: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:51.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:51 smithi132 bash[18607]: cluster 2024-04-25T20:07:50.238290+0000 mgr.x (mgr.14254) 327 : cluster [DBG] pgmap v265: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:51.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:51 smithi132 bash[18607]: audit 2024-04-25T20:07:50.789127+0000 mon.a (mon.0) 437 : audit [DBG] from='client.? 172.17.0.2:0/2365182437' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:07:51.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:51 smithi132 bash[18607]: audit 2024-04-25T20:07:50.789127+0000 mon.a (mon.0) 437 : audit [DBG] from='client.? 172.17.0.2:0/2365182437' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:07:53.500 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:07:53.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:53 smithi132 bash[18607]: cluster 2024-04-25T20:07:52.239690+0000 mgr.x (mgr.14254) 328 : cluster [DBG] pgmap v266: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:53.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:53 smithi132 bash[18607]: cluster 2024-04-25T20:07:52.239690+0000 mgr.x (mgr.14254) 328 : cluster [DBG] pgmap v266: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:55.514 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:55 smithi132 bash[18607]: cluster 2024-04-25T20:07:54.241560+0000 mgr.x (mgr.14254) 329 : cluster [DBG] pgmap v267: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:55.514 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:55 smithi132 bash[18607]: cluster 2024-04-25T20:07:54.241560+0000 mgr.x (mgr.14254) 329 : cluster [DBG] pgmap v267: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:56.203 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:07:56.204 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:07:56.879 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:07:57.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:57 smithi132 bash[18607]: audit 2024-04-25T20:07:56.209427+0000 mgr.x (mgr.14254) 330 : audit [DBG] from='client.14446 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:07:57.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:57 smithi132 bash[18607]: audit 2024-04-25T20:07:56.209427+0000 mgr.x (mgr.14254) 330 : audit [DBG] from='client.14446 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:07:57.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:57 smithi132 bash[18607]: cluster 2024-04-25T20:07:56.242212+0000 mgr.x (mgr.14254) 331 : cluster [DBG] pgmap v268: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:57.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:57 smithi132 bash[18607]: cluster 2024-04-25T20:07:56.242212+0000 mgr.x (mgr.14254) 331 : cluster [DBG] pgmap v268: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:57.880 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:07:59.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:59 smithi132 bash[18607]: cluster 2024-04-25T20:07:58.244038+0000 mgr.x (mgr.14254) 332 : cluster [DBG] pgmap v269: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:07:59.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:07:59 smithi132 bash[18607]: cluster 2024-04-25T20:07:58.244038+0000 mgr.x (mgr.14254) 332 : cluster [DBG] pgmap v269: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:01.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:01 smithi132 bash[18607]: cluster 2024-04-25T20:08:00.245560+0000 mgr.x (mgr.14254) 333 : cluster [DBG] pgmap v270: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:01.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:01 smithi132 bash[18607]: cluster 2024-04-25T20:08:00.245560+0000 mgr.x (mgr.14254) 333 : cluster [DBG] pgmap v270: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:02.637 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:08:03.695 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:03 smithi132 bash[18607]: cluster 2024-04-25T20:08:02.247120+0000 mgr.x (mgr.14254) 334 : cluster [DBG] pgmap v271: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:03.695 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:03 smithi132 bash[18607]: cluster 2024-04-25T20:08:02.247120+0000 mgr.x (mgr.14254) 334 : cluster [DBG] pgmap v271: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:05.460 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:08:05.461 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:08:05.671 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:05 smithi132 bash[18607]: cluster 2024-04-25T20:08:04.249282+0000 mgr.x (mgr.14254) 335 : cluster [DBG] pgmap v272: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:05.671 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:05 smithi132 bash[18607]: cluster 2024-04-25T20:08:04.249282+0000 mgr.x (mgr.14254) 335 : cluster [DBG] pgmap v272: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:06.156 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:08:06.514 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:06 smithi132 bash[18607]: audit 2024-04-25T20:08:05.466349+0000 mgr.x (mgr.14254) 336 : audit [DBG] from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:08:06.514 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:06 smithi132 bash[18607]: audit 2024-04-25T20:08:05.466349+0000 mgr.x (mgr.14254) 336 : audit [DBG] from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:08:07.157 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:08:07.513 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:07 smithi132 bash[18607]: cluster 2024-04-25T20:08:06.250388+0000 mgr.x (mgr.14254) 337 : cluster [DBG] pgmap v273: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:07.514 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:07 smithi132 bash[18607]: cluster 2024-04-25T20:08:06.250388+0000 mgr.x (mgr.14254) 337 : cluster [DBG] pgmap v273: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:09.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:09 smithi132 bash[18607]: cluster 2024-04-25T20:08:08.252153+0000 mgr.x (mgr.14254) 338 : cluster [DBG] pgmap v274: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:09.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:09 smithi132 bash[18607]: cluster 2024-04-25T20:08:08.252153+0000 mgr.x (mgr.14254) 338 : cluster [DBG] pgmap v274: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:11.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:11 smithi132 bash[18607]: cluster 2024-04-25T20:08:10.253564+0000 mgr.x (mgr.14254) 339 : cluster [DBG] pgmap v275: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:11.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:11 smithi132 bash[18607]: cluster 2024-04-25T20:08:10.253564+0000 mgr.x (mgr.14254) 339 : cluster [DBG] pgmap v275: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:11.918 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:08:13.763 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:13 smithi132 bash[18607]: cluster 2024-04-25T20:08:12.254933+0000 mgr.x (mgr.14254) 340 : cluster [DBG] pgmap v276: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:13.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:13 smithi132 bash[18607]: cluster 2024-04-25T20:08:12.254933+0000 mgr.x (mgr.14254) 340 : cluster [DBG] pgmap v276: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:14.668 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:08:14.669 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:08:15.260 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:08:15.513 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:15 smithi132 bash[18607]: cluster 2024-04-25T20:08:14.256741+0000 mgr.x (mgr.14254) 341 : cluster [DBG] pgmap v277: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:15.513 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:15 smithi132 bash[18607]: cluster 2024-04-25T20:08:14.256741+0000 mgr.x (mgr.14254) 341 : cluster [DBG] pgmap v277: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:16.261 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:08:16.763 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:16 smithi132 bash[18607]: audit 2024-04-25T20:08:14.674018+0000 mgr.x (mgr.14254) 342 : audit [DBG] from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:08:16.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:16 smithi132 bash[18607]: audit 2024-04-25T20:08:14.674018+0000 mgr.x (mgr.14254) 342 : audit [DBG] from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:08:17.763 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:17 smithi132 bash[18607]: cluster 2024-04-25T20:08:16.258079+0000 mgr.x (mgr.14254) 343 : cluster [DBG] pgmap v278: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:17.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:17 smithi132 bash[18607]: cluster 2024-04-25T20:08:16.258079+0000 mgr.x (mgr.14254) 343 : cluster [DBG] pgmap v278: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:19.763 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:19 smithi132 bash[18607]: cluster 2024-04-25T20:08:18.259843+0000 mgr.x (mgr.14254) 344 : cluster [DBG] pgmap v279: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:19.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:19 smithi132 bash[18607]: cluster 2024-04-25T20:08:18.259843+0000 mgr.x (mgr.14254) 344 : cluster [DBG] pgmap v279: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:21.027 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:08:21.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:21 smithi132 bash[18607]: cluster 2024-04-25T20:08:20.261267+0000 mgr.x (mgr.14254) 345 : cluster [DBG] pgmap v280: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:21.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:21 smithi132 bash[18607]: cluster 2024-04-25T20:08:20.261267+0000 mgr.x (mgr.14254) 345 : cluster [DBG] pgmap v280: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:23.716 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:08:23.716 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:08:23.730 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:23 smithi132 bash[18607]: cluster 2024-04-25T20:08:22.262780+0000 mgr.x (mgr.14254) 346 : cluster [DBG] pgmap v281: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:23.730 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:23 smithi132 bash[18607]: cluster 2024-04-25T20:08:22.262780+0000 mgr.x (mgr.14254) 346 : cluster [DBG] pgmap v281: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:24.346 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:08:25.348 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:08:25.763 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:25 smithi132 bash[18607]: audit 2024-04-25T20:08:23.722357+0000 mgr.x (mgr.14254) 347 : audit [DBG] from='client.14452 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:08:25.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:25 smithi132 bash[18607]: audit 2024-04-25T20:08:23.722357+0000 mgr.x (mgr.14254) 347 : audit [DBG] from='client.14452 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:08:25.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:25 smithi132 bash[18607]: cluster 2024-04-25T20:08:24.264675+0000 mgr.x (mgr.14254) 348 : cluster [DBG] pgmap v282: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:25.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:25 smithi132 bash[18607]: cluster 2024-04-25T20:08:24.264675+0000 mgr.x (mgr.14254) 348 : cluster [DBG] pgmap v282: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:27.763 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:27 smithi132 bash[18607]: cluster 2024-04-25T20:08:26.266038+0000 mgr.x (mgr.14254) 349 : cluster [DBG] pgmap v283: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:27.763 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:27 smithi132 bash[18607]: cluster 2024-04-25T20:08:26.266038+0000 mgr.x (mgr.14254) 349 : cluster [DBG] pgmap v283: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:29.763 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:29 smithi132 bash[18607]: cluster 2024-04-25T20:08:28.267884+0000 mgr.x (mgr.14254) 350 : cluster [DBG] pgmap v284: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:29.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:29 smithi132 bash[18607]: cluster 2024-04-25T20:08:28.267884+0000 mgr.x (mgr.14254) 350 : cluster [DBG] pgmap v284: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:30.106 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:08:31.514 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:31 smithi132 bash[18607]: cluster 2024-04-25T20:08:30.269362+0000 mgr.x (mgr.14254) 351 : cluster [DBG] pgmap v285: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:31.514 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:31 smithi132 bash[18607]: cluster 2024-04-25T20:08:30.269362+0000 mgr.x (mgr.14254) 351 : cluster [DBG] pgmap v285: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:32.810 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:08:32.810 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:08:33.504 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:08:33.763 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:33 smithi132 bash[18607]: cluster 2024-04-25T20:08:32.270794+0000 mgr.x (mgr.14254) 352 : cluster [DBG] pgmap v286: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:33.763 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:33 smithi132 bash[18607]: cluster 2024-04-25T20:08:32.270794+0000 mgr.x (mgr.14254) 352 : cluster [DBG] pgmap v286: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:34.506 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:08:34.763 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:34 smithi132 bash[18607]: audit 2024-04-25T20:08:32.818967+0000 mgr.x (mgr.14254) 353 : audit [DBG] from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:08:34.763 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:34 smithi132 bash[18607]: audit 2024-04-25T20:08:32.818967+0000 mgr.x (mgr.14254) 353 : audit [DBG] from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:08:35.763 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:35 smithi132 bash[18607]: cluster 2024-04-25T20:08:34.272685+0000 mgr.x (mgr.14254) 354 : cluster [DBG] pgmap v287: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:35.763 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:35 smithi132 bash[18607]: cluster 2024-04-25T20:08:34.272685+0000 mgr.x (mgr.14254) 354 : cluster [DBG] pgmap v287: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:37.763 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:37 smithi132 bash[18607]: cluster 2024-04-25T20:08:36.274099+0000 mgr.x (mgr.14254) 355 : cluster [DBG] pgmap v288: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:37.763 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:37 smithi132 bash[18607]: cluster 2024-04-25T20:08:36.274099+0000 mgr.x (mgr.14254) 355 : cluster [DBG] pgmap v288: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:39.272 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:08:39.763 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:39 smithi132 bash[18607]: cluster 2024-04-25T20:08:38.275923+0000 mgr.x (mgr.14254) 356 : cluster [DBG] pgmap v289: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:39.763 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:39 smithi132 bash[18607]: cluster 2024-04-25T20:08:38.275923+0000 mgr.x (mgr.14254) 356 : cluster [DBG] pgmap v289: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:41.682 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:41 smithi132 bash[18607]: cluster 2024-04-25T20:08:40.276896+0000 mgr.x (mgr.14254) 357 : cluster [DBG] pgmap v290: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:41.682 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:41 smithi132 bash[18607]: cluster 2024-04-25T20:08:40.276896+0000 mgr.x (mgr.14254) 357 : cluster [DBG] pgmap v290: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:42.083 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:08:42.083 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:08:42.713 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:08:43.714 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:08:43.725 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:43 smithi132 bash[18607]: audit 2024-04-25T20:08:42.089382+0000 mgr.x (mgr.14254) 358 : audit [DBG] from='client.14456 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:08:43.725 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:43 smithi132 bash[18607]: audit 2024-04-25T20:08:42.089382+0000 mgr.x (mgr.14254) 358 : audit [DBG] from='client.14456 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:08:43.725 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:43 smithi132 bash[18607]: cluster 2024-04-25T20:08:42.278298+0000 mgr.x (mgr.14254) 359 : cluster [DBG] pgmap v291: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:43.726 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:43 smithi132 bash[18607]: cluster 2024-04-25T20:08:42.278298+0000 mgr.x (mgr.14254) 359 : cluster [DBG] pgmap v291: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:45.763 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:45 smithi132 bash[18607]: cluster 2024-04-25T20:08:44.280139+0000 mgr.x (mgr.14254) 360 : cluster [DBG] pgmap v292: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:45.763 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:45 smithi132 bash[18607]: cluster 2024-04-25T20:08:44.280139+0000 mgr.x (mgr.14254) 360 : cluster [DBG] pgmap v292: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:47.763 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:47 smithi132 bash[18607]: cluster 2024-04-25T20:08:46.281573+0000 mgr.x (mgr.14254) 361 : cluster [DBG] pgmap v293: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:47.763 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:47 smithi132 bash[18607]: cluster 2024-04-25T20:08:46.281573+0000 mgr.x (mgr.14254) 361 : cluster [DBG] pgmap v293: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:48.475 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:08:49.596 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:49 smithi132 bash[18607]: cluster 2024-04-25T20:08:48.283404+0000 mgr.x (mgr.14254) 362 : cluster [DBG] pgmap v294: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:49.597 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:49 smithi132 bash[18607]: cluster 2024-04-25T20:08:48.283404+0000 mgr.x (mgr.14254) 362 : cluster [DBG] pgmap v294: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:51.139 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:08:51.139 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:08:51.752 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:51 smithi132 bash[18607]: cluster 2024-04-25T20:08:50.284869+0000 mgr.x (mgr.14254) 363 : cluster [DBG] pgmap v295: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:51.752 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:51 smithi132 bash[18607]: cluster 2024-04-25T20:08:50.284869+0000 mgr.x (mgr.14254) 363 : cluster [DBG] pgmap v295: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:51.753 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:08:52.754 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:08:52.763 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:52 smithi132 bash[18607]: audit 2024-04-25T20:08:51.145127+0000 mgr.x (mgr.14254) 364 : audit [DBG] from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:08:52.763 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:52 smithi132 bash[18607]: audit 2024-04-25T20:08:51.145127+0000 mgr.x (mgr.14254) 364 : audit [DBG] from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:08:53.763 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:53 smithi132 bash[18607]: cluster 2024-04-25T20:08:52.286258+0000 mgr.x (mgr.14254) 365 : cluster [DBG] pgmap v296: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:53.763 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:53 smithi132 bash[18607]: cluster 2024-04-25T20:08:52.286258+0000 mgr.x (mgr.14254) 365 : cluster [DBG] pgmap v296: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:55.763 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:55 smithi132 bash[18607]: cluster 2024-04-25T20:08:54.288125+0000 mgr.x (mgr.14254) 366 : cluster [DBG] pgmap v297: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:55.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:55 smithi132 bash[18607]: cluster 2024-04-25T20:08:54.288125+0000 mgr.x (mgr.14254) 366 : cluster [DBG] pgmap v297: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:57.512 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:08:57.763 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:57 smithi132 bash[18607]: cluster 2024-04-25T20:08:56.289554+0000 mgr.x (mgr.14254) 367 : cluster [DBG] pgmap v298: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:57.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:57 smithi132 bash[18607]: cluster 2024-04-25T20:08:56.289554+0000 mgr.x (mgr.14254) 367 : cluster [DBG] pgmap v298: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:59.513 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:59 smithi132 bash[18607]: cluster 2024-04-25T20:08:58.291369+0000 mgr.x (mgr.14254) 368 : cluster [DBG] pgmap v299: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:08:59.514 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:08:59 smithi132 bash[18607]: cluster 2024-04-25T20:08:58.291369+0000 mgr.x (mgr.14254) 368 : cluster [DBG] pgmap v299: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:00.218 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:09:00.218 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:09:00.930 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:09:01.514 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:01 smithi132 bash[18607]: audit 2024-04-25T20:09:00.223929+0000 mgr.x (mgr.14254) 369 : audit [DBG] from='client.14460 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:09:01.514 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:01 smithi132 bash[18607]: audit 2024-04-25T20:09:00.223929+0000 mgr.x (mgr.14254) 369 : audit [DBG] from='client.14460 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:09:01.514 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:01 smithi132 bash[18607]: cluster 2024-04-25T20:09:00.292096+0000 mgr.x (mgr.14254) 370 : cluster [DBG] pgmap v300: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:01.514 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:01 smithi132 bash[18607]: cluster 2024-04-25T20:09:00.292096+0000 mgr.x (mgr.14254) 370 : cluster [DBG] pgmap v300: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:01.514 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:01 smithi132 bash[18607]: audit 2024-04-25T20:09:01.173516+0000 mon.a (mon.0) 438 : audit [DBG] from='client.? 172.17.0.2:0/838153227' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:09:01.514 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:01 smithi132 bash[18607]: audit 2024-04-25T20:09:01.173516+0000 mon.a (mon.0) 438 : audit [DBG] from='client.? 172.17.0.2:0/838153227' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:09:01.934 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:09:03.763 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:03 smithi132 bash[18607]: cluster 2024-04-25T20:09:02.293489+0000 mgr.x (mgr.14254) 371 : cluster [DBG] pgmap v301: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:03.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:03 smithi132 bash[18607]: cluster 2024-04-25T20:09:02.293489+0000 mgr.x (mgr.14254) 371 : cluster [DBG] pgmap v301: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:05.763 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:05 smithi132 bash[18607]: cluster 2024-04-25T20:09:04.295349+0000 mgr.x (mgr.14254) 372 : cluster [DBG] pgmap v302: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:05.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:05 smithi132 bash[18607]: cluster 2024-04-25T20:09:04.295349+0000 mgr.x (mgr.14254) 372 : cluster [DBG] pgmap v302: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:06.695 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:09:07.763 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:07 smithi132 bash[18607]: cluster 2024-04-25T20:09:06.296726+0000 mgr.x (mgr.14254) 373 : cluster [DBG] pgmap v303: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:07.763 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:07 smithi132 bash[18607]: cluster 2024-04-25T20:09:06.296726+0000 mgr.x (mgr.14254) 373 : cluster [DBG] pgmap v303: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:09.445 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:09:09.445 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:09:09.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:09 smithi132 bash[18607]: cluster 2024-04-25T20:09:08.298466+0000 mgr.x (mgr.14254) 374 : cluster [DBG] pgmap v304: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:09.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:09 smithi132 bash[18607]: cluster 2024-04-25T20:09:08.298466+0000 mgr.x (mgr.14254) 374 : cluster [DBG] pgmap v304: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:10.088 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:09:10.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:10 smithi132 bash[18607]: audit 2024-04-25T20:09:09.450873+0000 mgr.x (mgr.14254) 375 : audit [DBG] from='client.14464 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:09:10.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:10 smithi132 bash[18607]: audit 2024-04-25T20:09:09.450873+0000 mgr.x (mgr.14254) 375 : audit [DBG] from='client.14464 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:09:11.089 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:09:11.469 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:11 smithi132 bash[18607]: cluster 2024-04-25T20:09:10.299896+0000 mgr.x (mgr.14254) 376 : cluster [DBG] pgmap v305: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:11.469 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:11 smithi132 bash[18607]: cluster 2024-04-25T20:09:10.299896+0000 mgr.x (mgr.14254) 376 : cluster [DBG] pgmap v305: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:13.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:13 smithi132 bash[18607]: cluster 2024-04-25T20:09:12.301315+0000 mgr.x (mgr.14254) 377 : cluster [DBG] pgmap v306: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:13.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:13 smithi132 bash[18607]: cluster 2024-04-25T20:09:12.301315+0000 mgr.x (mgr.14254) 377 : cluster [DBG] pgmap v306: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:15.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:15 smithi132 bash[18607]: cluster 2024-04-25T20:09:14.303152+0000 mgr.x (mgr.14254) 378 : cluster [DBG] pgmap v307: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:15.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:15 smithi132 bash[18607]: cluster 2024-04-25T20:09:14.303152+0000 mgr.x (mgr.14254) 378 : cluster [DBG] pgmap v307: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:15.851 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:09:17.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:17 smithi132 bash[18607]: cluster 2024-04-25T20:09:16.304721+0000 mgr.x (mgr.14254) 379 : cluster [DBG] pgmap v308: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:17.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:17 smithi132 bash[18607]: cluster 2024-04-25T20:09:16.304721+0000 mgr.x (mgr.14254) 379 : cluster [DBG] pgmap v308: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:18.708 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:09:18.708 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:09:19.334 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:09:19.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:19 smithi132 bash[18607]: cluster 2024-04-25T20:09:18.305720+0000 mgr.x (mgr.14254) 380 : cluster [DBG] pgmap v309: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:19.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:19 smithi132 bash[18607]: cluster 2024-04-25T20:09:18.305720+0000 mgr.x (mgr.14254) 380 : cluster [DBG] pgmap v309: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:20.334 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:09:20.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:20 smithi132 bash[18607]: audit 2024-04-25T20:09:18.714087+0000 mgr.x (mgr.14254) 381 : audit [DBG] from='client.14466 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:09:20.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:20 smithi132 bash[18607]: audit 2024-04-25T20:09:18.714087+0000 mgr.x (mgr.14254) 381 : audit [DBG] from='client.14466 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:09:21.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:21 smithi132 bash[18607]: cluster 2024-04-25T20:09:20.307234+0000 mgr.x (mgr.14254) 382 : cluster [DBG] pgmap v310: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:21.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:21 smithi132 bash[18607]: cluster 2024-04-25T20:09:20.307234+0000 mgr.x (mgr.14254) 382 : cluster [DBG] pgmap v310: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:23.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:23 smithi132 bash[18607]: cluster 2024-04-25T20:09:22.308831+0000 mgr.x (mgr.14254) 383 : cluster [DBG] pgmap v311: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:23.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:23 smithi132 bash[18607]: cluster 2024-04-25T20:09:22.308831+0000 mgr.x (mgr.14254) 383 : cluster [DBG] pgmap v311: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:25.094 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:09:25.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:25 smithi132 bash[18607]: cluster 2024-04-25T20:09:24.310684+0000 mgr.x (mgr.14254) 384 : cluster [DBG] pgmap v312: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:25.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:25 smithi132 bash[18607]: cluster 2024-04-25T20:09:24.310684+0000 mgr.x (mgr.14254) 384 : cluster [DBG] pgmap v312: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:27.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:27 smithi132 bash[18607]: cluster 2024-04-25T20:09:26.312125+0000 mgr.x (mgr.14254) 385 : cluster [DBG] pgmap v313: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:27.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:27 smithi132 bash[18607]: cluster 2024-04-25T20:09:26.312125+0000 mgr.x (mgr.14254) 385 : cluster [DBG] pgmap v313: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:27.818 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:09:27.818 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:09:28.432 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:09:29.434 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:09:29.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:29 smithi132 bash[18607]: audit 2024-04-25T20:09:27.824126+0000 mgr.x (mgr.14254) 386 : audit [DBG] from='client.14468 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:09:29.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:29 smithi132 bash[18607]: audit 2024-04-25T20:09:27.824126+0000 mgr.x (mgr.14254) 386 : audit [DBG] from='client.14468 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:09:29.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:29 smithi132 bash[18607]: cluster 2024-04-25T20:09:28.313990+0000 mgr.x (mgr.14254) 387 : cluster [DBG] pgmap v314: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:29.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:29 smithi132 bash[18607]: cluster 2024-04-25T20:09:28.313990+0000 mgr.x (mgr.14254) 387 : cluster [DBG] pgmap v314: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:31.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:31 smithi132 bash[18607]: cluster 2024-04-25T20:09:30.315411+0000 mgr.x (mgr.14254) 388 : cluster [DBG] pgmap v315: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:31.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:31 smithi132 bash[18607]: cluster 2024-04-25T20:09:30.315411+0000 mgr.x (mgr.14254) 388 : cluster [DBG] pgmap v315: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:33.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:33 smithi132 bash[18607]: cluster 2024-04-25T20:09:32.316863+0000 mgr.x (mgr.14254) 389 : cluster [DBG] pgmap v316: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:33.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:33 smithi132 bash[18607]: cluster 2024-04-25T20:09:32.316863+0000 mgr.x (mgr.14254) 389 : cluster [DBG] pgmap v316: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:34.197 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:09:35.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:35 smithi132 bash[18607]: cluster 2024-04-25T20:09:34.318804+0000 mgr.x (mgr.14254) 390 : cluster [DBG] pgmap v317: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:35.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:35 smithi132 bash[18607]: cluster 2024-04-25T20:09:34.318804+0000 mgr.x (mgr.14254) 390 : cluster [DBG] pgmap v317: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:36.929 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:09:36.929 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:09:37.550 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:09:37.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:37 smithi132 bash[18607]: cluster 2024-04-25T20:09:36.320195+0000 mgr.x (mgr.14254) 391 : cluster [DBG] pgmap v318: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:37.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:37 smithi132 bash[18607]: cluster 2024-04-25T20:09:36.320195+0000 mgr.x (mgr.14254) 391 : cluster [DBG] pgmap v318: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:38.551 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:09:38.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:38 smithi132 bash[18607]: audit 2024-04-25T20:09:36.936725+0000 mgr.x (mgr.14254) 392 : audit [DBG] from='client.14470 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:09:38.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:38 smithi132 bash[18607]: audit 2024-04-25T20:09:36.936725+0000 mgr.x (mgr.14254) 392 : audit [DBG] from='client.14470 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:09:39.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:39 smithi132 bash[18607]: cluster 2024-04-25T20:09:38.321973+0000 mgr.x (mgr.14254) 393 : cluster [DBG] pgmap v319: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:39.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:39 smithi132 bash[18607]: cluster 2024-04-25T20:09:38.321973+0000 mgr.x (mgr.14254) 393 : cluster [DBG] pgmap v319: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:41.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:41 smithi132 bash[18607]: cluster 2024-04-25T20:09:40.323344+0000 mgr.x (mgr.14254) 394 : cluster [DBG] pgmap v320: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:41.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:41 smithi132 bash[18607]: cluster 2024-04-25T20:09:40.323344+0000 mgr.x (mgr.14254) 394 : cluster [DBG] pgmap v320: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:43.306 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:09:43.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:43 smithi132 bash[18607]: cluster 2024-04-25T20:09:42.324744+0000 mgr.x (mgr.14254) 395 : cluster [DBG] pgmap v321: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:43.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:43 smithi132 bash[18607]: cluster 2024-04-25T20:09:42.324744+0000 mgr.x (mgr.14254) 395 : cluster [DBG] pgmap v321: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:45.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:45 smithi132 bash[18607]: cluster 2024-04-25T20:09:44.325892+0000 mgr.x (mgr.14254) 396 : cluster [DBG] pgmap v322: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:45.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:45 smithi132 bash[18607]: cluster 2024-04-25T20:09:44.325892+0000 mgr.x (mgr.14254) 396 : cluster [DBG] pgmap v322: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:46.022 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:09:46.022 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:09:46.692 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:09:47.694 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:09:47.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:47 smithi132 bash[18607]: audit 2024-04-25T20:09:46.027305+0000 mgr.x (mgr.14254) 397 : audit [DBG] from='client.14472 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:09:47.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:47 smithi132 bash[18607]: audit 2024-04-25T20:09:46.027305+0000 mgr.x (mgr.14254) 397 : audit [DBG] from='client.14472 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:09:47.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:47 smithi132 bash[18607]: cluster 2024-04-25T20:09:46.327302+0000 mgr.x (mgr.14254) 398 : cluster [DBG] pgmap v323: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:47.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:47 smithi132 bash[18607]: cluster 2024-04-25T20:09:46.327302+0000 mgr.x (mgr.14254) 398 : cluster [DBG] pgmap v323: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:49.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:49 smithi132 bash[18607]: cluster 2024-04-25T20:09:48.329272+0000 mgr.x (mgr.14254) 399 : cluster [DBG] pgmap v324: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:49.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:49 smithi132 bash[18607]: cluster 2024-04-25T20:09:48.329272+0000 mgr.x (mgr.14254) 399 : cluster [DBG] pgmap v324: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:52.014 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:51 smithi132 bash[18607]: cluster 2024-04-25T20:09:50.330714+0000 mgr.x (mgr.14254) 400 : cluster [DBG] pgmap v325: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:52.014 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:51 smithi132 bash[18607]: cluster 2024-04-25T20:09:50.330714+0000 mgr.x (mgr.14254) 400 : cluster [DBG] pgmap v325: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:52.451 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:09:54.014 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:53 smithi132 bash[18607]: cluster 2024-04-25T20:09:52.332241+0000 mgr.x (mgr.14254) 401 : cluster [DBG] pgmap v326: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:54.014 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:53 smithi132 bash[18607]: cluster 2024-04-25T20:09:52.332241+0000 mgr.x (mgr.14254) 401 : cluster [DBG] pgmap v326: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:55.106 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:09:55.106 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:09:55.795 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:55 smithi132 bash[18607]: cluster 2024-04-25T20:09:54.334287+0000 mgr.x (mgr.14254) 402 : cluster [DBG] pgmap v327: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:55.795 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:55 smithi132 bash[18607]: cluster 2024-04-25T20:09:54.334287+0000 mgr.x (mgr.14254) 402 : cluster [DBG] pgmap v327: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:55.796 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:09:56.797 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:09:56.808 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:56 smithi132 bash[18607]: audit 2024-04-25T20:09:55.111789+0000 mgr.x (mgr.14254) 403 : audit [DBG] from='client.14474 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:09:56.808 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:56 smithi132 bash[18607]: audit 2024-04-25T20:09:55.111789+0000 mgr.x (mgr.14254) 403 : audit [DBG] from='client.14474 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:09:58.014 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:57 smithi132 bash[18607]: cluster 2024-04-25T20:09:56.335731+0000 mgr.x (mgr.14254) 404 : cluster [DBG] pgmap v328: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:09:58.014 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:57 smithi132 bash[18607]: cluster 2024-04-25T20:09:56.335731+0000 mgr.x (mgr.14254) 404 : cluster [DBG] pgmap v328: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:00.014 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:59 smithi132 bash[18607]: cluster 2024-04-25T20:09:58.337609+0000 mgr.x (mgr.14254) 405 : cluster [DBG] pgmap v329: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:00.014 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:09:59 smithi132 bash[18607]: cluster 2024-04-25T20:09:58.337609+0000 mgr.x (mgr.14254) 405 : cluster [DBG] pgmap v329: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:01.562 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:10:02.014 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:01 smithi132 bash[18607]: cluster 2024-04-25T20:10:00.339029+0000 mgr.x (mgr.14254) 406 : cluster [DBG] pgmap v330: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:02.014 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:01 smithi132 bash[18607]: cluster 2024-04-25T20:10:00.339029+0000 mgr.x (mgr.14254) 406 : cluster [DBG] pgmap v330: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:03.786 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:03 smithi132 bash[18607]: cluster 2024-04-25T20:10:02.340444+0000 mgr.x (mgr.14254) 407 : cluster [DBG] pgmap v331: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:03.786 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:03 smithi132 bash[18607]: cluster 2024-04-25T20:10:02.340444+0000 mgr.x (mgr.14254) 407 : cluster [DBG] pgmap v331: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:04.158 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:10:04.159 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:10:04.801 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:10:05.014 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:04 smithi132 bash[18607]: audit 2024-04-25T20:10:04.165659+0000 mgr.x (mgr.14254) 408 : audit [DBG] from='client.14476 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:10:05.014 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:04 smithi132 bash[18607]: audit 2024-04-25T20:10:04.165659+0000 mgr.x (mgr.14254) 408 : audit [DBG] from='client.14476 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:10:05.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:04 smithi132 bash[18607]: cluster 2024-04-25T20:10:04.342448+0000 mgr.x (mgr.14254) 409 : cluster [DBG] pgmap v332: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:05.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:04 smithi132 bash[18607]: cluster 2024-04-25T20:10:04.342448+0000 mgr.x (mgr.14254) 409 : cluster [DBG] pgmap v332: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:05.802 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:10:07.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:07 smithi132 bash[18607]: cluster 2024-04-25T20:10:06.343977+0000 mgr.x (mgr.14254) 410 : cluster [DBG] pgmap v333: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:07.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:07 smithi132 bash[18607]: cluster 2024-04-25T20:10:06.343977+0000 mgr.x (mgr.14254) 410 : cluster [DBG] pgmap v333: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:09.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:09 smithi132 bash[18607]: cluster 2024-04-25T20:10:08.345864+0000 mgr.x (mgr.14254) 411 : cluster [DBG] pgmap v334: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:09.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:09 smithi132 bash[18607]: cluster 2024-04-25T20:10:08.345864+0000 mgr.x (mgr.14254) 411 : cluster [DBG] pgmap v334: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:10.564 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:10:11.701 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:11 smithi132 bash[18607]: cluster 2024-04-25T20:10:10.347267+0000 mgr.x (mgr.14254) 412 : cluster [DBG] pgmap v335: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:11.701 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:11 smithi132 bash[18607]: cluster 2024-04-25T20:10:10.347267+0000 mgr.x (mgr.14254) 412 : cluster [DBG] pgmap v335: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:12.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:12 smithi132 bash[18607]: audit 2024-04-25T20:10:11.551758+0000 mon.a (mon.0) 439 : audit [DBG] from='client.? 172.17.0.2:0/2296666981' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:10:12.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:12 smithi132 bash[18607]: audit 2024-04-25T20:10:11.551758+0000 mon.a (mon.0) 439 : audit [DBG] from='client.? 172.17.0.2:0/2296666981' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:10:13.382 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:10:13.382 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:10:13.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:13 smithi132 bash[18607]: cluster 2024-04-25T20:10:12.348768+0000 mgr.x (mgr.14254) 413 : cluster [DBG] pgmap v336: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:13.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:13 smithi132 bash[18607]: cluster 2024-04-25T20:10:12.348768+0000 mgr.x (mgr.14254) 413 : cluster [DBG] pgmap v336: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:14.054 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:10:14.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:14 smithi132 bash[18607]: audit 2024-04-25T20:10:13.387245+0000 mgr.x (mgr.14254) 414 : audit [DBG] from='client.14480 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:10:14.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:14 smithi132 bash[18607]: audit 2024-04-25T20:10:13.387245+0000 mgr.x (mgr.14254) 414 : audit [DBG] from='client.14480 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:10:15.055 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:10:15.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:15 smithi132 bash[18607]: cluster 2024-04-25T20:10:14.350534+0000 mgr.x (mgr.14254) 415 : cluster [DBG] pgmap v337: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:15.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:15 smithi132 bash[18607]: cluster 2024-04-25T20:10:14.350534+0000 mgr.x (mgr.14254) 415 : cluster [DBG] pgmap v337: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:17.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:17 smithi132 bash[18607]: cluster 2024-04-25T20:10:16.351923+0000 mgr.x (mgr.14254) 416 : cluster [DBG] pgmap v338: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:17.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:17 smithi132 bash[18607]: cluster 2024-04-25T20:10:16.351923+0000 mgr.x (mgr.14254) 416 : cluster [DBG] pgmap v338: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:19.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:19 smithi132 bash[18607]: cluster 2024-04-25T20:10:18.353721+0000 mgr.x (mgr.14254) 417 : cluster [DBG] pgmap v339: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:19.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:19 smithi132 bash[18607]: cluster 2024-04-25T20:10:18.353721+0000 mgr.x (mgr.14254) 417 : cluster [DBG] pgmap v339: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:19.815 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:10:21.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:21 smithi132 bash[18607]: cluster 2024-04-25T20:10:20.355112+0000 mgr.x (mgr.14254) 418 : cluster [DBG] pgmap v340: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:21.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:21 smithi132 bash[18607]: cluster 2024-04-25T20:10:20.355112+0000 mgr.x (mgr.14254) 418 : cluster [DBG] pgmap v340: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:22.717 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:10:22.717 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:10:23.375 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:10:23.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:23 smithi132 bash[18607]: cluster 2024-04-25T20:10:22.355986+0000 mgr.x (mgr.14254) 419 : cluster [DBG] pgmap v341: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:23.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:23 smithi132 bash[18607]: cluster 2024-04-25T20:10:22.355986+0000 mgr.x (mgr.14254) 419 : cluster [DBG] pgmap v341: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:24.377 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:10:24.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:24 smithi132 bash[18607]: audit 2024-04-25T20:10:22.721819+0000 mgr.x (mgr.14254) 420 : audit [DBG] from='client.14482 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:10:24.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:24 smithi132 bash[18607]: audit 2024-04-25T20:10:22.721819+0000 mgr.x (mgr.14254) 420 : audit [DBG] from='client.14482 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:10:25.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:25 smithi132 bash[18607]: cluster 2024-04-25T20:10:24.357785+0000 mgr.x (mgr.14254) 421 : cluster [DBG] pgmap v342: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:25.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:25 smithi132 bash[18607]: cluster 2024-04-25T20:10:24.357785+0000 mgr.x (mgr.14254) 421 : cluster [DBG] pgmap v342: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:27.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:27 smithi132 bash[18607]: cluster 2024-04-25T20:10:26.359154+0000 mgr.x (mgr.14254) 422 : cluster [DBG] pgmap v343: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:27.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:27 smithi132 bash[18607]: cluster 2024-04-25T20:10:26.359154+0000 mgr.x (mgr.14254) 422 : cluster [DBG] pgmap v343: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:29.132 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:10:29.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:29 smithi132 bash[18607]: cluster 2024-04-25T20:10:28.360955+0000 mgr.x (mgr.14254) 423 : cluster [DBG] pgmap v344: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:29.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:29 smithi132 bash[18607]: cluster 2024-04-25T20:10:28.360955+0000 mgr.x (mgr.14254) 423 : cluster [DBG] pgmap v344: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:31.764 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:31 smithi132 bash[18607]: cluster 2024-04-25T20:10:30.361770+0000 mgr.x (mgr.14254) 424 : cluster [DBG] pgmap v345: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:31.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:31 smithi132 bash[18607]: cluster 2024-04-25T20:10:30.361770+0000 mgr.x (mgr.14254) 424 : cluster [DBG] pgmap v345: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:31.964 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:10:31.965 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:10:32.677 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:10:33.678 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:10:33.689 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:33 smithi132 bash[18607]: audit 2024-04-25T20:10:31.972408+0000 mgr.x (mgr.14254) 425 : audit [DBG] from='client.14484 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:10:33.689 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:33 smithi132 bash[18607]: audit 2024-04-25T20:10:31.972408+0000 mgr.x (mgr.14254) 425 : audit [DBG] from='client.14484 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:10:33.689 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:33 smithi132 bash[18607]: cluster 2024-04-25T20:10:32.363161+0000 mgr.x (mgr.14254) 426 : cluster [DBG] pgmap v346: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:33.690 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:33 smithi132 bash[18607]: cluster 2024-04-25T20:10:32.363161+0000 mgr.x (mgr.14254) 426 : cluster [DBG] pgmap v346: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:35.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:35 smithi132 bash[18607]: cluster 2024-04-25T20:10:34.365122+0000 mgr.x (mgr.14254) 427 : cluster [DBG] pgmap v347: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:35.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:35 smithi132 bash[18607]: cluster 2024-04-25T20:10:34.365122+0000 mgr.x (mgr.14254) 427 : cluster [DBG] pgmap v347: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:37.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:37 smithi132 bash[18607]: cluster 2024-04-25T20:10:36.366520+0000 mgr.x (mgr.14254) 428 : cluster [DBG] pgmap v348: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:37.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:37 smithi132 bash[18607]: cluster 2024-04-25T20:10:36.366520+0000 mgr.x (mgr.14254) 428 : cluster [DBG] pgmap v348: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:38.439 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:10:39.579 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:39 smithi132 bash[18607]: cluster 2024-04-25T20:10:38.368358+0000 mgr.x (mgr.14254) 429 : cluster [DBG] pgmap v349: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:39.579 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:39 smithi132 bash[18607]: cluster 2024-04-25T20:10:38.368358+0000 mgr.x (mgr.14254) 429 : cluster [DBG] pgmap v349: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:41.098 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:10:41.098 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:10:41.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:41 smithi132 bash[18607]: cluster 2024-04-25T20:10:40.369916+0000 mgr.x (mgr.14254) 430 : cluster [DBG] pgmap v350: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:41.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:41 smithi132 bash[18607]: cluster 2024-04-25T20:10:40.369916+0000 mgr.x (mgr.14254) 430 : cluster [DBG] pgmap v350: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:41.772 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:10:42.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:42 smithi132 bash[18607]: audit 2024-04-25T20:10:41.105609+0000 mgr.x (mgr.14254) 431 : audit [DBG] from='client.14486 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:10:42.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:42 smithi132 bash[18607]: audit 2024-04-25T20:10:41.105609+0000 mgr.x (mgr.14254) 431 : audit [DBG] from='client.14486 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:10:42.772 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:10:43.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:43 smithi132 bash[18607]: cluster 2024-04-25T20:10:42.371462+0000 mgr.x (mgr.14254) 432 : cluster [DBG] pgmap v351: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:43.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:43 smithi132 bash[18607]: cluster 2024-04-25T20:10:42.371462+0000 mgr.x (mgr.14254) 432 : cluster [DBG] pgmap v351: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:45.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:45 smithi132 bash[18607]: cluster 2024-04-25T20:10:44.373399+0000 mgr.x (mgr.14254) 433 : cluster [DBG] pgmap v352: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:45.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:45 smithi132 bash[18607]: cluster 2024-04-25T20:10:44.373399+0000 mgr.x (mgr.14254) 433 : cluster [DBG] pgmap v352: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:47.529 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:10:47.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:47 smithi132 bash[18607]: cluster 2024-04-25T20:10:46.374963+0000 mgr.x (mgr.14254) 434 : cluster [DBG] pgmap v353: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:47.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:47 smithi132 bash[18607]: cluster 2024-04-25T20:10:46.374963+0000 mgr.x (mgr.14254) 434 : cluster [DBG] pgmap v353: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:49.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:49 smithi132 bash[18607]: cluster 2024-04-25T20:10:48.375979+0000 mgr.x (mgr.14254) 435 : cluster [DBG] pgmap v354: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:49.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:49 smithi132 bash[18607]: cluster 2024-04-25T20:10:48.375979+0000 mgr.x (mgr.14254) 435 : cluster [DBG] pgmap v354: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:50.224 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:10:50.225 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:10:50.827 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:10:51.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:51 smithi132 bash[18607]: audit 2024-04-25T20:10:50.229325+0000 mgr.x (mgr.14254) 436 : audit [DBG] from='client.14488 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:10:51.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:51 smithi132 bash[18607]: audit 2024-04-25T20:10:50.229325+0000 mgr.x (mgr.14254) 436 : audit [DBG] from='client.14488 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:10:51.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:51 smithi132 bash[18607]: cluster 2024-04-25T20:10:50.377433+0000 mgr.x (mgr.14254) 437 : cluster [DBG] pgmap v355: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:51.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:51 smithi132 bash[18607]: cluster 2024-04-25T20:10:50.377433+0000 mgr.x (mgr.14254) 437 : cluster [DBG] pgmap v355: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:51.828 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:10:53.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:53 smithi132 bash[18607]: cluster 2024-04-25T20:10:52.378806+0000 mgr.x (mgr.14254) 438 : cluster [DBG] pgmap v356: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:53.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:53 smithi132 bash[18607]: cluster 2024-04-25T20:10:52.378806+0000 mgr.x (mgr.14254) 438 : cluster [DBG] pgmap v356: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:55.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:55 smithi132 bash[18607]: cluster 2024-04-25T20:10:54.380599+0000 mgr.x (mgr.14254) 439 : cluster [DBG] pgmap v357: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:55.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:55 smithi132 bash[18607]: cluster 2024-04-25T20:10:54.380599+0000 mgr.x (mgr.14254) 439 : cluster [DBG] pgmap v357: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:56.639 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:10:57.677 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:57 smithi132 bash[18607]: cluster 2024-04-25T20:10:56.381971+0000 mgr.x (mgr.14254) 440 : cluster [DBG] pgmap v358: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:57.677 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:57 smithi132 bash[18607]: cluster 2024-04-25T20:10:56.381971+0000 mgr.x (mgr.14254) 440 : cluster [DBG] pgmap v358: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:59.311 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:10:59.311 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:10:59.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:59 smithi132 bash[18607]: cluster 2024-04-25T20:10:58.384434+0000 mgr.x (mgr.14254) 441 : cluster [DBG] pgmap v359: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:59.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:10:59 smithi132 bash[18607]: cluster 2024-04-25T20:10:58.384434+0000 mgr.x (mgr.14254) 441 : cluster [DBG] pgmap v359: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:10:59.941 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:11:00.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:00 smithi132 bash[18607]: audit 2024-04-25T20:10:59.317792+0000 mgr.x (mgr.14254) 442 : audit [DBG] from='client.14490 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:11:00.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:00 smithi132 bash[18607]: audit 2024-04-25T20:10:59.317792+0000 mgr.x (mgr.14254) 442 : audit [DBG] from='client.14490 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:11:00.942 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:11:01.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:01 smithi132 bash[18607]: cluster 2024-04-25T20:11:00.385863+0000 mgr.x (mgr.14254) 443 : cluster [DBG] pgmap v360: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:01.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:01 smithi132 bash[18607]: cluster 2024-04-25T20:11:00.385863+0000 mgr.x (mgr.14254) 443 : cluster [DBG] pgmap v360: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:03.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:03 smithi132 bash[18607]: cluster 2024-04-25T20:11:02.387252+0000 mgr.x (mgr.14254) 444 : cluster [DBG] pgmap v361: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:03.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:03 smithi132 bash[18607]: cluster 2024-04-25T20:11:02.387252+0000 mgr.x (mgr.14254) 444 : cluster [DBG] pgmap v361: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:05.705 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:11:05.729 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:05 smithi132 bash[18607]: cluster 2024-04-25T20:11:04.389164+0000 mgr.x (mgr.14254) 445 : cluster [DBG] pgmap v362: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:05.729 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:05 smithi132 bash[18607]: cluster 2024-04-25T20:11:04.389164+0000 mgr.x (mgr.14254) 445 : cluster [DBG] pgmap v362: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:07.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:07 smithi132 bash[18607]: cluster 2024-04-25T20:11:06.390613+0000 mgr.x (mgr.14254) 446 : cluster [DBG] pgmap v363: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:07.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:07 smithi132 bash[18607]: cluster 2024-04-25T20:11:06.390613+0000 mgr.x (mgr.14254) 446 : cluster [DBG] pgmap v363: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:08.337 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:11:08.337 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:11:08.975 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:11:09.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:09 smithi132 bash[18607]: audit 2024-04-25T20:11:08.341905+0000 mgr.x (mgr.14254) 447 : audit [DBG] from='client.14492 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:11:09.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:09 smithi132 bash[18607]: audit 2024-04-25T20:11:08.341905+0000 mgr.x (mgr.14254) 447 : audit [DBG] from='client.14492 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:11:09.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:09 smithi132 bash[18607]: cluster 2024-04-25T20:11:08.391479+0000 mgr.x (mgr.14254) 448 : cluster [DBG] pgmap v364: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:09.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:09 smithi132 bash[18607]: cluster 2024-04-25T20:11:08.391479+0000 mgr.x (mgr.14254) 448 : cluster [DBG] pgmap v364: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:09.976 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:11:11.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:11 smithi132 bash[18607]: cluster 2024-04-25T20:11:10.392880+0000 mgr.x (mgr.14254) 449 : cluster [DBG] pgmap v365: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:11.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:11 smithi132 bash[18607]: cluster 2024-04-25T20:11:10.392880+0000 mgr.x (mgr.14254) 449 : cluster [DBG] pgmap v365: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:13.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:13 smithi132 bash[18607]: cluster 2024-04-25T20:11:12.394268+0000 mgr.x (mgr.14254) 450 : cluster [DBG] pgmap v366: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:13.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:13 smithi132 bash[18607]: cluster 2024-04-25T20:11:12.394268+0000 mgr.x (mgr.14254) 450 : cluster [DBG] pgmap v366: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:14.734 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:11:15.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:15 smithi132 bash[18607]: cluster 2024-04-25T20:11:14.396108+0000 mgr.x (mgr.14254) 451 : cluster [DBG] pgmap v367: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:15.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:15 smithi132 bash[18607]: cluster 2024-04-25T20:11:14.396108+0000 mgr.x (mgr.14254) 451 : cluster [DBG] pgmap v367: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:17.466 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:11:17.473 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:11:17.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:17 smithi132 bash[18607]: cluster 2024-04-25T20:11:16.397082+0000 mgr.x (mgr.14254) 452 : cluster [DBG] pgmap v368: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:17.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:17 smithi132 bash[18607]: cluster 2024-04-25T20:11:16.397082+0000 mgr.x (mgr.14254) 452 : cluster [DBG] pgmap v368: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:18.161 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:11:18.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:18 smithi132 bash[18607]: audit 2024-04-25T20:11:17.473454+0000 mgr.x (mgr.14254) 453 : audit [DBG] from='client.14494 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:11:18.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:18 smithi132 bash[18607]: audit 2024-04-25T20:11:17.473454+0000 mgr.x (mgr.14254) 453 : audit [DBG] from='client.14494 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:11:19.163 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:11:19.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:19 smithi132 bash[18607]: cluster 2024-04-25T20:11:18.398929+0000 mgr.x (mgr.14254) 454 : cluster [DBG] pgmap v369: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:19.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:19 smithi132 bash[18607]: cluster 2024-04-25T20:11:18.398929+0000 mgr.x (mgr.14254) 454 : cluster [DBG] pgmap v369: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:21.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:21 smithi132 bash[18607]: cluster 2024-04-25T20:11:20.399778+0000 mgr.x (mgr.14254) 455 : cluster [DBG] pgmap v370: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:21.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:21 smithi132 bash[18607]: cluster 2024-04-25T20:11:20.399778+0000 mgr.x (mgr.14254) 455 : cluster [DBG] pgmap v370: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:22.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:22 smithi132 bash[18607]: audit 2024-04-25T20:11:21.932341+0000 mon.a (mon.0) 440 : audit [DBG] from='client.? 172.17.0.2:0/1652433226' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:11:22.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:22 smithi132 bash[18607]: audit 2024-04-25T20:11:21.932341+0000 mon.a (mon.0) 440 : audit [DBG] from='client.? 172.17.0.2:0/1652433226' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:11:23.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:23 smithi132 bash[18607]: cluster 2024-04-25T20:11:22.401836+0000 mgr.x (mgr.14254) 456 : cluster [DBG] pgmap v371: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:23.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:23 smithi132 bash[18607]: cluster 2024-04-25T20:11:22.401836+0000 mgr.x (mgr.14254) 456 : cluster [DBG] pgmap v371: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:23.924 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:11:25.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:25 smithi132 bash[18607]: cluster 2024-04-25T20:11:24.403790+0000 mgr.x (mgr.14254) 457 : cluster [DBG] pgmap v372: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:25.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:25 smithi132 bash[18607]: cluster 2024-04-25T20:11:24.403790+0000 mgr.x (mgr.14254) 457 : cluster [DBG] pgmap v372: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:26.703 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:11:26.703 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:11:27.384 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:11:27.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:27 smithi132 bash[18607]: cluster 2024-04-25T20:11:26.405210+0000 mgr.x (mgr.14254) 458 : cluster [DBG] pgmap v373: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:27.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:27 smithi132 bash[18607]: cluster 2024-04-25T20:11:26.405210+0000 mgr.x (mgr.14254) 458 : cluster [DBG] pgmap v373: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:28.385 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:11:28.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:28 smithi132 bash[18607]: audit 2024-04-25T20:11:26.709013+0000 mgr.x (mgr.14254) 459 : audit [DBG] from='client.14498 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:11:28.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:28 smithi132 bash[18607]: audit 2024-04-25T20:11:26.709013+0000 mgr.x (mgr.14254) 459 : audit [DBG] from='client.14498 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:11:29.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:29 smithi132 bash[18607]: cluster 2024-04-25T20:11:28.407085+0000 mgr.x (mgr.14254) 460 : cluster [DBG] pgmap v374: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:29.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:29 smithi132 bash[18607]: cluster 2024-04-25T20:11:28.407085+0000 mgr.x (mgr.14254) 460 : cluster [DBG] pgmap v374: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:31.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:31 smithi132 bash[18607]: cluster 2024-04-25T20:11:30.408436+0000 mgr.x (mgr.14254) 461 : cluster [DBG] pgmap v375: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:31.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:31 smithi132 bash[18607]: cluster 2024-04-25T20:11:30.408436+0000 mgr.x (mgr.14254) 461 : cluster [DBG] pgmap v375: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:33.142 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:11:33.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:33 smithi132 bash[18607]: cluster 2024-04-25T20:11:32.409842+0000 mgr.x (mgr.14254) 462 : cluster [DBG] pgmap v376: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:33.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:33 smithi132 bash[18607]: cluster 2024-04-25T20:11:32.409842+0000 mgr.x (mgr.14254) 462 : cluster [DBG] pgmap v376: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:35.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:35 smithi132 bash[18607]: cluster 2024-04-25T20:11:34.411655+0000 mgr.x (mgr.14254) 463 : cluster [DBG] pgmap v377: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:35.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:35 smithi132 bash[18607]: cluster 2024-04-25T20:11:34.411655+0000 mgr.x (mgr.14254) 463 : cluster [DBG] pgmap v377: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:35.904 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:11:35.905 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:11:36.551 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:11:37.552 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:11:37.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:37 smithi132 bash[18607]: audit 2024-04-25T20:11:35.909164+0000 mgr.x (mgr.14254) 464 : audit [DBG] from='client.14500 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:11:37.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:37 smithi132 bash[18607]: audit 2024-04-25T20:11:35.909164+0000 mgr.x (mgr.14254) 464 : audit [DBG] from='client.14500 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:11:37.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:37 smithi132 bash[18607]: cluster 2024-04-25T20:11:36.413094+0000 mgr.x (mgr.14254) 465 : cluster [DBG] pgmap v378: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:37.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:37 smithi132 bash[18607]: cluster 2024-04-25T20:11:36.413094+0000 mgr.x (mgr.14254) 465 : cluster [DBG] pgmap v378: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:40.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:39 smithi132 bash[18607]: cluster 2024-04-25T20:11:38.414951+0000 mgr.x (mgr.14254) 466 : cluster [DBG] pgmap v379: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:40.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:39 smithi132 bash[18607]: cluster 2024-04-25T20:11:38.414951+0000 mgr.x (mgr.14254) 466 : cluster [DBG] pgmap v379: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:42.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:41 smithi132 bash[18607]: cluster 2024-04-25T20:11:40.416478+0000 mgr.x (mgr.14254) 467 : cluster [DBG] pgmap v380: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:42.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:41 smithi132 bash[18607]: cluster 2024-04-25T20:11:40.416478+0000 mgr.x (mgr.14254) 467 : cluster [DBG] pgmap v380: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:42.311 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:11:43.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:43 smithi132 bash[18607]: cluster 2024-04-25T20:11:42.418023+0000 mgr.x (mgr.14254) 468 : cluster [DBG] pgmap v381: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:43.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:43 smithi132 bash[18607]: cluster 2024-04-25T20:11:42.418023+0000 mgr.x (mgr.14254) 468 : cluster [DBG] pgmap v381: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:45.183 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:11:45.183 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:11:45.834 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:45 smithi132 bash[18607]: cluster 2024-04-25T20:11:44.419914+0000 mgr.x (mgr.14254) 469 : cluster [DBG] pgmap v382: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:45.835 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:45 smithi132 bash[18607]: cluster 2024-04-25T20:11:44.419914+0000 mgr.x (mgr.14254) 469 : cluster [DBG] pgmap v382: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:45.835 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:11:46.836 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:11:46.847 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:46 smithi132 bash[18607]: audit 2024-04-25T20:11:45.187304+0000 mgr.x (mgr.14254) 470 : audit [DBG] from='client.14502 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:11:46.847 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:46 smithi132 bash[18607]: audit 2024-04-25T20:11:45.187304+0000 mgr.x (mgr.14254) 470 : audit [DBG] from='client.14502 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:11:48.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:47 smithi132 bash[18607]: cluster 2024-04-25T20:11:46.421352+0000 mgr.x (mgr.14254) 471 : cluster [DBG] pgmap v383: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:48.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:47 smithi132 bash[18607]: cluster 2024-04-25T20:11:46.421352+0000 mgr.x (mgr.14254) 471 : cluster [DBG] pgmap v383: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:50.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:49 smithi132 bash[18607]: cluster 2024-04-25T20:11:48.423164+0000 mgr.x (mgr.14254) 472 : cluster [DBG] pgmap v384: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:50.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:49 smithi132 bash[18607]: cluster 2024-04-25T20:11:48.423164+0000 mgr.x (mgr.14254) 472 : cluster [DBG] pgmap v384: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:51.592 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:11:52.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:51 smithi132 bash[18607]: cluster 2024-04-25T20:11:50.424561+0000 mgr.x (mgr.14254) 473 : cluster [DBG] pgmap v385: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:52.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:51 smithi132 bash[18607]: cluster 2024-04-25T20:11:50.424561+0000 mgr.x (mgr.14254) 473 : cluster [DBG] pgmap v385: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:53.784 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:53 smithi132 bash[18607]: cluster 2024-04-25T20:11:52.425337+0000 mgr.x (mgr.14254) 474 : cluster [DBG] pgmap v386: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:53.784 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:53 smithi132 bash[18607]: cluster 2024-04-25T20:11:52.425337+0000 mgr.x (mgr.14254) 474 : cluster [DBG] pgmap v386: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:54.156 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:11:54.156 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:11:54.815 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:11:55.816 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:11:55.828 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:55 smithi132 bash[18607]: audit 2024-04-25T20:11:54.160822+0000 mgr.x (mgr.14254) 475 : audit [DBG] from='client.14504 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:11:55.828 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:55 smithi132 bash[18607]: audit 2024-04-25T20:11:54.160822+0000 mgr.x (mgr.14254) 475 : audit [DBG] from='client.14504 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:11:55.828 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:55 smithi132 bash[18607]: cluster 2024-04-25T20:11:54.427214+0000 mgr.x (mgr.14254) 476 : cluster [DBG] pgmap v387: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:55.829 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:55 smithi132 bash[18607]: cluster 2024-04-25T20:11:54.427214+0000 mgr.x (mgr.14254) 476 : cluster [DBG] pgmap v387: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:58.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:57 smithi132 bash[18607]: cluster 2024-04-25T20:11:56.428659+0000 mgr.x (mgr.14254) 477 : cluster [DBG] pgmap v388: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:11:58.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:57 smithi132 bash[18607]: cluster 2024-04-25T20:11:56.428659+0000 mgr.x (mgr.14254) 477 : cluster [DBG] pgmap v388: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:00.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:59 smithi132 bash[18607]: cluster 2024-04-25T20:11:58.430518+0000 mgr.x (mgr.14254) 478 : cluster [DBG] pgmap v389: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:00.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:11:59 smithi132 bash[18607]: cluster 2024-04-25T20:11:58.430518+0000 mgr.x (mgr.14254) 478 : cluster [DBG] pgmap v389: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:00.572 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:12:01.683 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:01 smithi132 bash[18607]: cluster 2024-04-25T20:12:00.431976+0000 mgr.x (mgr.14254) 479 : cluster [DBG] pgmap v390: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:01.683 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:01 smithi132 bash[18607]: cluster 2024-04-25T20:12:00.431976+0000 mgr.x (mgr.14254) 479 : cluster [DBG] pgmap v390: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:03.303 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:12:03.303 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:12:03.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:03 smithi132 bash[18607]: cluster 2024-04-25T20:12:02.433461+0000 mgr.x (mgr.14254) 480 : cluster [DBG] pgmap v391: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:03.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:03 smithi132 bash[18607]: cluster 2024-04-25T20:12:02.433461+0000 mgr.x (mgr.14254) 480 : cluster [DBG] pgmap v391: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:03.925 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:12:04.926 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:12:04.937 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:04 smithi132 bash[18607]: audit 2024-04-25T20:12:03.307355+0000 mgr.x (mgr.14254) 481 : audit [DBG] from='client.14506 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:12:04.937 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:04 smithi132 bash[18607]: audit 2024-04-25T20:12:03.307355+0000 mgr.x (mgr.14254) 481 : audit [DBG] from='client.14506 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:12:06.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:05 smithi132 bash[18607]: cluster 2024-04-25T20:12:04.435480+0000 mgr.x (mgr.14254) 482 : cluster [DBG] pgmap v392: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:06.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:05 smithi132 bash[18607]: cluster 2024-04-25T20:12:04.435480+0000 mgr.x (mgr.14254) 482 : cluster [DBG] pgmap v392: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:08.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:07 smithi132 bash[18607]: cluster 2024-04-25T20:12:06.437070+0000 mgr.x (mgr.14254) 483 : cluster [DBG] pgmap v393: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:08.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:07 smithi132 bash[18607]: cluster 2024-04-25T20:12:06.437070+0000 mgr.x (mgr.14254) 483 : cluster [DBG] pgmap v393: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:09.684 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:12:10.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:09 smithi132 bash[18607]: cluster 2024-04-25T20:12:08.439025+0000 mgr.x (mgr.14254) 484 : cluster [DBG] pgmap v394: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:10.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:09 smithi132 bash[18607]: cluster 2024-04-25T20:12:08.439025+0000 mgr.x (mgr.14254) 484 : cluster [DBG] pgmap v394: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:11.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:11 smithi132 bash[18607]: cluster 2024-04-25T20:12:10.440620+0000 mgr.x (mgr.14254) 485 : cluster [DBG] pgmap v395: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:11.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:11 smithi132 bash[18607]: cluster 2024-04-25T20:12:10.440620+0000 mgr.x (mgr.14254) 485 : cluster [DBG] pgmap v395: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:12.509 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:12:12.509 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:12:13.162 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:12:14.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:13 smithi132 bash[18607]: cluster 2024-04-25T20:12:12.442074+0000 mgr.x (mgr.14254) 486 : cluster [DBG] pgmap v396: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:14.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:13 smithi132 bash[18607]: cluster 2024-04-25T20:12:12.442074+0000 mgr.x (mgr.14254) 486 : cluster [DBG] pgmap v396: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:14.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:13 smithi132 bash[18607]: audit 2024-04-25T20:12:12.513621+0000 mgr.x (mgr.14254) 487 : audit [DBG] from='client.14508 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:12:14.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:13 smithi132 bash[18607]: audit 2024-04-25T20:12:12.513621+0000 mgr.x (mgr.14254) 487 : audit [DBG] from='client.14508 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:12:14.163 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:12:16.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:15 smithi132 bash[18607]: cluster 2024-04-25T20:12:14.443834+0000 mgr.x (mgr.14254) 488 : cluster [DBG] pgmap v397: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:16.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:15 smithi132 bash[18607]: cluster 2024-04-25T20:12:14.443834+0000 mgr.x (mgr.14254) 488 : cluster [DBG] pgmap v397: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:18.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:17 smithi132 bash[18607]: cluster 2024-04-25T20:12:16.445315+0000 mgr.x (mgr.14254) 489 : cluster [DBG] pgmap v398: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:18.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:17 smithi132 bash[18607]: cluster 2024-04-25T20:12:16.445315+0000 mgr.x (mgr.14254) 489 : cluster [DBG] pgmap v398: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:18.920 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:12:18.941 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:18 smithi132 bash[18607]: cluster 2024-04-25T20:12:18.447132+0000 mgr.x (mgr.14254) 490 : cluster [DBG] pgmap v399: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:18.941 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:18 smithi132 bash[18607]: cluster 2024-04-25T20:12:18.447132+0000 mgr.x (mgr.14254) 490 : cluster [DBG] pgmap v399: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:21.608 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:12:21.608 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:12:21.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:21 smithi132 bash[18607]: cluster 2024-04-25T20:12:20.448536+0000 mgr.x (mgr.14254) 491 : cluster [DBG] pgmap v400: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:21.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:21 smithi132 bash[18607]: cluster 2024-04-25T20:12:20.448536+0000 mgr.x (mgr.14254) 491 : cluster [DBG] pgmap v400: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:22.251 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:12:23.252 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:12:23.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:23 smithi132 bash[18607]: audit 2024-04-25T20:12:21.614220+0000 mgr.x (mgr.14254) 492 : audit [DBG] from='client.14510 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:12:23.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:23 smithi132 bash[18607]: audit 2024-04-25T20:12:21.614220+0000 mgr.x (mgr.14254) 492 : audit [DBG] from='client.14510 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:12:23.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:23 smithi132 bash[18607]: cluster 2024-04-25T20:12:22.449924+0000 mgr.x (mgr.14254) 493 : cluster [DBG] pgmap v401: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:23.515 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:23 smithi132 bash[18607]: cluster 2024-04-25T20:12:22.449924+0000 mgr.x (mgr.14254) 493 : cluster [DBG] pgmap v401: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:25.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:25 smithi132 bash[18607]: cluster 2024-04-25T20:12:24.451754+0000 mgr.x (mgr.14254) 494 : cluster [DBG] pgmap v402: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:25.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:25 smithi132 bash[18607]: cluster 2024-04-25T20:12:24.451754+0000 mgr.x (mgr.14254) 494 : cluster [DBG] pgmap v402: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:27.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:27 smithi132 bash[18607]: cluster 2024-04-25T20:12:26.453204+0000 mgr.x (mgr.14254) 495 : cluster [DBG] pgmap v403: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:27.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:27 smithi132 bash[18607]: cluster 2024-04-25T20:12:26.453204+0000 mgr.x (mgr.14254) 495 : cluster [DBG] pgmap v403: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:28.009 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:12:29.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:29 smithi132 bash[18607]: cluster 2024-04-25T20:12:28.455072+0000 mgr.x (mgr.14254) 496 : cluster [DBG] pgmap v404: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:29.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:29 smithi132 bash[18607]: cluster 2024-04-25T20:12:28.455072+0000 mgr.x (mgr.14254) 496 : cluster [DBG] pgmap v404: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:30.694 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:12:30.694 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:12:31.366 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:12:31.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:31 smithi132 bash[18607]: cluster 2024-04-25T20:12:30.456034+0000 mgr.x (mgr.14254) 497 : cluster [DBG] pgmap v405: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:31.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:31 smithi132 bash[18607]: cluster 2024-04-25T20:12:30.456034+0000 mgr.x (mgr.14254) 497 : cluster [DBG] pgmap v405: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:32.366 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:12:32.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:32 smithi132 bash[18607]: audit 2024-04-25T20:12:30.698778+0000 mgr.x (mgr.14254) 498 : audit [DBG] from='client.14512 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:12:32.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:32 smithi132 bash[18607]: audit 2024-04-25T20:12:30.698778+0000 mgr.x (mgr.14254) 498 : audit [DBG] from='client.14512 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:12:32.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:32 smithi132 bash[18607]: audit 2024-04-25T20:12:32.307690+0000 mon.a (mon.0) 441 : audit [DBG] from='client.? 172.17.0.2:0/3352292464' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:12:32.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:32 smithi132 bash[18607]: audit 2024-04-25T20:12:32.307690+0000 mon.a (mon.0) 441 : audit [DBG] from='client.? 172.17.0.2:0/3352292464' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:12:33.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:33 smithi132 bash[18607]: cluster 2024-04-25T20:12:32.456964+0000 mgr.x (mgr.14254) 499 : cluster [DBG] pgmap v406: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:33.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:33 smithi132 bash[18607]: cluster 2024-04-25T20:12:32.456964+0000 mgr.x (mgr.14254) 499 : cluster [DBG] pgmap v406: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:35.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:35 smithi132 bash[18607]: cluster 2024-04-25T20:12:34.458724+0000 mgr.x (mgr.14254) 500 : cluster [DBG] pgmap v407: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:35.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:35 smithi132 bash[18607]: cluster 2024-04-25T20:12:34.458724+0000 mgr.x (mgr.14254) 500 : cluster [DBG] pgmap v407: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:37.117 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:12:37.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:37 smithi132 bash[18607]: cluster 2024-04-25T20:12:36.460122+0000 mgr.x (mgr.14254) 501 : cluster [DBG] pgmap v408: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:37.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:37 smithi132 bash[18607]: cluster 2024-04-25T20:12:36.460122+0000 mgr.x (mgr.14254) 501 : cluster [DBG] pgmap v408: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:39.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:39 smithi132 bash[18607]: cluster 2024-04-25T20:12:38.462048+0000 mgr.x (mgr.14254) 502 : cluster [DBG] pgmap v409: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:39.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:39 smithi132 bash[18607]: cluster 2024-04-25T20:12:38.462048+0000 mgr.x (mgr.14254) 502 : cluster [DBG] pgmap v409: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:39.804 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:12:39.804 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:12:40.489 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:12:41.490 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:12:41.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:41 smithi132 bash[18607]: audit 2024-04-25T20:12:39.808466+0000 mgr.x (mgr.14254) 503 : audit [DBG] from='client.14516 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:12:41.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:41 smithi132 bash[18607]: audit 2024-04-25T20:12:39.808466+0000 mgr.x (mgr.14254) 503 : audit [DBG] from='client.14516 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:12:41.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:41 smithi132 bash[18607]: cluster 2024-04-25T20:12:40.463551+0000 mgr.x (mgr.14254) 504 : cluster [DBG] pgmap v410: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:41.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:41 smithi132 bash[18607]: cluster 2024-04-25T20:12:40.463551+0000 mgr.x (mgr.14254) 504 : cluster [DBG] pgmap v410: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:44.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:43 smithi132 bash[18607]: cluster 2024-04-25T20:12:42.465048+0000 mgr.x (mgr.14254) 505 : cluster [DBG] pgmap v411: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:44.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:43 smithi132 bash[18607]: cluster 2024-04-25T20:12:42.465048+0000 mgr.x (mgr.14254) 505 : cluster [DBG] pgmap v411: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:46.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:45 smithi132 bash[18607]: cluster 2024-04-25T20:12:44.466935+0000 mgr.x (mgr.14254) 506 : cluster [DBG] pgmap v412: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:46.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:45 smithi132 bash[18607]: cluster 2024-04-25T20:12:44.466935+0000 mgr.x (mgr.14254) 506 : cluster [DBG] pgmap v412: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:46.247 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:12:47.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:47 smithi132 bash[18607]: cluster 2024-04-25T20:12:46.468395+0000 mgr.x (mgr.14254) 507 : cluster [DBG] pgmap v413: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:47.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:47 smithi132 bash[18607]: cluster 2024-04-25T20:12:46.468395+0000 mgr.x (mgr.14254) 507 : cluster [DBG] pgmap v413: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:48.989 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:12:48.989 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:12:49.618 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:12:50.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:49 smithi132 bash[18607]: cluster 2024-04-25T20:12:48.470248+0000 mgr.x (mgr.14254) 508 : cluster [DBG] pgmap v414: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:50.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:49 smithi132 bash[18607]: cluster 2024-04-25T20:12:48.470248+0000 mgr.x (mgr.14254) 508 : cluster [DBG] pgmap v414: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:50.620 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:12:51.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:50 smithi132 bash[18607]: audit 2024-04-25T20:12:48.993241+0000 mgr.x (mgr.14254) 509 : audit [DBG] from='client.14518 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:12:51.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:50 smithi132 bash[18607]: audit 2024-04-25T20:12:48.993241+0000 mgr.x (mgr.14254) 509 : audit [DBG] from='client.14518 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:12:52.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:51 smithi132 bash[18607]: cluster 2024-04-25T20:12:50.471594+0000 mgr.x (mgr.14254) 510 : cluster [DBG] pgmap v415: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:52.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:51 smithi132 bash[18607]: cluster 2024-04-25T20:12:50.471594+0000 mgr.x (mgr.14254) 510 : cluster [DBG] pgmap v415: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:54.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:53 smithi132 bash[18607]: cluster 2024-04-25T20:12:52.473068+0000 mgr.x (mgr.14254) 511 : cluster [DBG] pgmap v416: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:54.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:53 smithi132 bash[18607]: cluster 2024-04-25T20:12:52.473068+0000 mgr.x (mgr.14254) 511 : cluster [DBG] pgmap v416: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:55.380 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:12:55.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:55 smithi132 bash[18607]: cluster 2024-04-25T20:12:54.474914+0000 mgr.x (mgr.14254) 512 : cluster [DBG] pgmap v417: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:55.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:55 smithi132 bash[18607]: cluster 2024-04-25T20:12:54.474914+0000 mgr.x (mgr.14254) 512 : cluster [DBG] pgmap v417: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:57.824 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:57 smithi132 bash[18607]: cluster 2024-04-25T20:12:56.475710+0000 mgr.x (mgr.14254) 513 : cluster [DBG] pgmap v418: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:57.825 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:57 smithi132 bash[18607]: cluster 2024-04-25T20:12:56.475710+0000 mgr.x (mgr.14254) 513 : cluster [DBG] pgmap v418: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:58.237 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:12:58.237 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:12:58.920 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:12:59.921 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:12:59.933 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:59 smithi132 bash[18607]: audit 2024-04-25T20:12:58.241219+0000 mgr.x (mgr.14254) 514 : audit [DBG] from='client.14520 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:12:59.933 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:59 smithi132 bash[18607]: audit 2024-04-25T20:12:58.241219+0000 mgr.x (mgr.14254) 514 : audit [DBG] from='client.14520 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:12:59.933 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:59 smithi132 bash[18607]: cluster 2024-04-25T20:12:58.477482+0000 mgr.x (mgr.14254) 515 : cluster [DBG] pgmap v419: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:12:59.933 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:12:59 smithi132 bash[18607]: cluster 2024-04-25T20:12:58.477482+0000 mgr.x (mgr.14254) 515 : cluster [DBG] pgmap v419: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:02.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:01 smithi132 bash[18607]: cluster 2024-04-25T20:13:00.478883+0000 mgr.x (mgr.14254) 516 : cluster [DBG] pgmap v420: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:02.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:01 smithi132 bash[18607]: cluster 2024-04-25T20:13:00.478883+0000 mgr.x (mgr.14254) 516 : cluster [DBG] pgmap v420: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:04.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:03 smithi132 bash[18607]: cluster 2024-04-25T20:13:02.480299+0000 mgr.x (mgr.14254) 517 : cluster [DBG] pgmap v421: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:04.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:03 smithi132 bash[18607]: cluster 2024-04-25T20:13:02.480299+0000 mgr.x (mgr.14254) 517 : cluster [DBG] pgmap v421: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:04.681 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:13:05.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:05 smithi132 bash[18607]: cluster 2024-04-25T20:13:04.482177+0000 mgr.x (mgr.14254) 518 : cluster [DBG] pgmap v422: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:05.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:05 smithi132 bash[18607]: cluster 2024-04-25T20:13:04.482177+0000 mgr.x (mgr.14254) 518 : cluster [DBG] pgmap v422: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:07.310 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:13:07.310 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:13:07.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:07 smithi132 bash[18607]: cluster 2024-04-25T20:13:06.483600+0000 mgr.x (mgr.14254) 519 : cluster [DBG] pgmap v423: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:07.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:07 smithi132 bash[18607]: cluster 2024-04-25T20:13:06.483600+0000 mgr.x (mgr.14254) 519 : cluster [DBG] pgmap v423: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:07.915 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:13:08.916 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:13:08.927 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:08 smithi132 bash[18607]: audit 2024-04-25T20:13:07.314222+0000 mgr.x (mgr.14254) 520 : audit [DBG] from='client.14522 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:13:08.927 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:08 smithi132 bash[18607]: audit 2024-04-25T20:13:07.314222+0000 mgr.x (mgr.14254) 520 : audit [DBG] from='client.14522 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:13:10.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:09 smithi132 bash[18607]: cluster 2024-04-25T20:13:08.485552+0000 mgr.x (mgr.14254) 521 : cluster [DBG] pgmap v424: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:10.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:09 smithi132 bash[18607]: cluster 2024-04-25T20:13:08.485552+0000 mgr.x (mgr.14254) 521 : cluster [DBG] pgmap v424: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:12.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:11 smithi132 bash[18607]: cluster 2024-04-25T20:13:10.486996+0000 mgr.x (mgr.14254) 522 : cluster [DBG] pgmap v425: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:12.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:11 smithi132 bash[18607]: cluster 2024-04-25T20:13:10.486996+0000 mgr.x (mgr.14254) 522 : cluster [DBG] pgmap v425: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:13.674 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:13:14.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:13 smithi132 bash[18607]: cluster 2024-04-25T20:13:12.488363+0000 mgr.x (mgr.14254) 523 : cluster [DBG] pgmap v426: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:14.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:13 smithi132 bash[18607]: cluster 2024-04-25T20:13:12.488363+0000 mgr.x (mgr.14254) 523 : cluster [DBG] pgmap v426: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:15.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:15 smithi132 bash[18607]: cluster 2024-04-25T20:13:14.490394+0000 mgr.x (mgr.14254) 524 : cluster [DBG] pgmap v427: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:15.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:15 smithi132 bash[18607]: cluster 2024-04-25T20:13:14.490394+0000 mgr.x (mgr.14254) 524 : cluster [DBG] pgmap v427: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:16.401 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:13:16.401 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:13:17.065 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:13:18.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:17 smithi132 bash[18607]: audit 2024-04-25T20:13:16.408092+0000 mgr.x (mgr.14254) 525 : audit [DBG] from='client.14524 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:13:18.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:17 smithi132 bash[18607]: audit 2024-04-25T20:13:16.408092+0000 mgr.x (mgr.14254) 525 : audit [DBG] from='client.14524 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:13:18.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:17 smithi132 bash[18607]: cluster 2024-04-25T20:13:16.491163+0000 mgr.x (mgr.14254) 526 : cluster [DBG] pgmap v428: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:18.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:17 smithi132 bash[18607]: cluster 2024-04-25T20:13:16.491163+0000 mgr.x (mgr.14254) 526 : cluster [DBG] pgmap v428: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:18.066 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:13:20.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:19 smithi132 bash[18607]: cluster 2024-04-25T20:13:18.493046+0000 mgr.x (mgr.14254) 527 : cluster [DBG] pgmap v429: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:20.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:19 smithi132 bash[18607]: cluster 2024-04-25T20:13:18.493046+0000 mgr.x (mgr.14254) 527 : cluster [DBG] pgmap v429: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:22.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:21 smithi132 bash[18607]: cluster 2024-04-25T20:13:20.494525+0000 mgr.x (mgr.14254) 528 : cluster [DBG] pgmap v430: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:22.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:21 smithi132 bash[18607]: cluster 2024-04-25T20:13:20.494525+0000 mgr.x (mgr.14254) 528 : cluster [DBG] pgmap v430: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:22.827 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:13:23.875 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:23 smithi132 bash[18607]: cluster 2024-04-25T20:13:22.495960+0000 mgr.x (mgr.14254) 529 : cluster [DBG] pgmap v431: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:23.875 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:23 smithi132 bash[18607]: cluster 2024-04-25T20:13:22.495960+0000 mgr.x (mgr.14254) 529 : cluster [DBG] pgmap v431: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:25.639 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:13:25.639 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:13:25.820 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:25 smithi132 bash[18607]: cluster 2024-04-25T20:13:24.497371+0000 mgr.x (mgr.14254) 530 : cluster [DBG] pgmap v432: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:25.820 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:25 smithi132 bash[18607]: cluster 2024-04-25T20:13:24.497371+0000 mgr.x (mgr.14254) 530 : cluster [DBG] pgmap v432: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:26.258 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:13:27.259 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:13:28.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:27 smithi132 bash[18607]: audit 2024-04-25T20:13:25.643510+0000 mgr.x (mgr.14254) 531 : audit [DBG] from='client.14526 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:13:28.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:27 smithi132 bash[18607]: audit 2024-04-25T20:13:25.643510+0000 mgr.x (mgr.14254) 531 : audit [DBG] from='client.14526 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:13:28.016 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:27 smithi132 bash[18607]: cluster 2024-04-25T20:13:26.498921+0000 mgr.x (mgr.14254) 532 : cluster [DBG] pgmap v433: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:28.016 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:27 smithi132 bash[18607]: cluster 2024-04-25T20:13:26.498921+0000 mgr.x (mgr.14254) 532 : cluster [DBG] pgmap v433: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:30.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:29 smithi132 bash[18607]: cluster 2024-04-25T20:13:28.500420+0000 mgr.x (mgr.14254) 533 : cluster [DBG] pgmap v434: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:30.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:29 smithi132 bash[18607]: cluster 2024-04-25T20:13:28.500420+0000 mgr.x (mgr.14254) 533 : cluster [DBG] pgmap v434: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:32.012 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:13:32.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:31 smithi132 bash[18607]: cluster 2024-04-25T20:13:30.501956+0000 mgr.x (mgr.14254) 534 : cluster [DBG] pgmap v435: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:32.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:31 smithi132 bash[18607]: cluster 2024-04-25T20:13:30.501956+0000 mgr.x (mgr.14254) 534 : cluster [DBG] pgmap v435: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:32.892 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:32 smithi132 bash[18607]: cluster 2024-04-25T20:13:32.503343+0000 mgr.x (mgr.14254) 535 : cluster [DBG] pgmap v436: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:32.892 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:32 smithi132 bash[18607]: cluster 2024-04-25T20:13:32.503343+0000 mgr.x (mgr.14254) 535 : cluster [DBG] pgmap v436: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:34.816 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:13:34.816 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:13:35.465 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:13:35.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:35 smithi132 bash[18607]: cluster 2024-04-25T20:13:34.504269+0000 mgr.x (mgr.14254) 536 : cluster [DBG] pgmap v437: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:35.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:35 smithi132 bash[18607]: cluster 2024-04-25T20:13:34.504269+0000 mgr.x (mgr.14254) 536 : cluster [DBG] pgmap v437: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:36.466 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:13:36.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:36 smithi132 bash[18607]: audit 2024-04-25T20:13:34.820361+0000 mgr.x (mgr.14254) 537 : audit [DBG] from='client.14528 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:13:36.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:36 smithi132 bash[18607]: audit 2024-04-25T20:13:34.820361+0000 mgr.x (mgr.14254) 537 : audit [DBG] from='client.14528 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:13:38.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:37 smithi132 bash[18607]: cluster 2024-04-25T20:13:36.505689+0000 mgr.x (mgr.14254) 538 : cluster [DBG] pgmap v438: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:38.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:37 smithi132 bash[18607]: cluster 2024-04-25T20:13:36.505689+0000 mgr.x (mgr.14254) 538 : cluster [DBG] pgmap v438: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:40.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:39 smithi132 bash[18607]: cluster 2024-04-25T20:13:38.507504+0000 mgr.x (mgr.14254) 539 : cluster [DBG] pgmap v439: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:40.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:39 smithi132 bash[18607]: cluster 2024-04-25T20:13:38.507504+0000 mgr.x (mgr.14254) 539 : cluster [DBG] pgmap v439: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:41.224 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:13:41.991 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:41 smithi132 bash[18607]: cluster 2024-04-25T20:13:40.508932+0000 mgr.x (mgr.14254) 540 : cluster [DBG] pgmap v440: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:41.991 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:41 smithi132 bash[18607]: cluster 2024-04-25T20:13:40.508932+0000 mgr.x (mgr.14254) 540 : cluster [DBG] pgmap v440: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:43.978 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:13:43.979 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:13:43.991 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:43 smithi132 bash[18607]: cluster 2024-04-25T20:13:42.510299+0000 mgr.x (mgr.14254) 541 : cluster [DBG] pgmap v441: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:43.992 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:43 smithi132 bash[18607]: cluster 2024-04-25T20:13:42.510299+0000 mgr.x (mgr.14254) 541 : cluster [DBG] pgmap v441: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:43.992 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:43 smithi132 bash[18607]: audit 2024-04-25T20:13:42.695460+0000 mon.a (mon.0) 442 : audit [DBG] from='client.? 172.17.0.2:0/132364975' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:13:43.992 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:43 smithi132 bash[18607]: audit 2024-04-25T20:13:42.695460+0000 mon.a (mon.0) 442 : audit [DBG] from='client.? 172.17.0.2:0/132364975' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:13:44.637 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:13:45.638 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:13:46.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:45 smithi132 bash[18607]: audit 2024-04-25T20:13:43.983105+0000 mgr.x (mgr.14254) 542 : audit [DBG] from='client.14532 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:13:46.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:45 smithi132 bash[18607]: audit 2024-04-25T20:13:43.983105+0000 mgr.x (mgr.14254) 542 : audit [DBG] from='client.14532 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:13:46.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:45 smithi132 bash[18607]: cluster 2024-04-25T20:13:44.512092+0000 mgr.x (mgr.14254) 543 : cluster [DBG] pgmap v442: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:46.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:45 smithi132 bash[18607]: cluster 2024-04-25T20:13:44.512092+0000 mgr.x (mgr.14254) 543 : cluster [DBG] pgmap v442: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:48.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:47 smithi132 bash[18607]: cluster 2024-04-25T20:13:46.513526+0000 mgr.x (mgr.14254) 544 : cluster [DBG] pgmap v443: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:48.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:47 smithi132 bash[18607]: cluster 2024-04-25T20:13:46.513526+0000 mgr.x (mgr.14254) 544 : cluster [DBG] pgmap v443: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:50.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:49 smithi132 bash[18607]: cluster 2024-04-25T20:13:48.515375+0000 mgr.x (mgr.14254) 545 : cluster [DBG] pgmap v444: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:50.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:49 smithi132 bash[18607]: cluster 2024-04-25T20:13:48.515375+0000 mgr.x (mgr.14254) 545 : cluster [DBG] pgmap v444: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:50.390 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:13:52.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:51 smithi132 bash[18607]: cluster 2024-04-25T20:13:50.516795+0000 mgr.x (mgr.14254) 546 : cluster [DBG] pgmap v445: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:52.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:51 smithi132 bash[18607]: cluster 2024-04-25T20:13:50.516795+0000 mgr.x (mgr.14254) 546 : cluster [DBG] pgmap v445: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:53.136 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:13:53.136 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:13:53.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:53 smithi132 bash[18607]: cluster 2024-04-25T20:13:52.518273+0000 mgr.x (mgr.14254) 547 : cluster [DBG] pgmap v446: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:53.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:53 smithi132 bash[18607]: cluster 2024-04-25T20:13:52.518273+0000 mgr.x (mgr.14254) 547 : cluster [DBG] pgmap v446: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:53.828 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:13:54.830 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:13:54.841 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:54 smithi132 bash[18607]: audit 2024-04-25T20:13:53.143552+0000 mgr.x (mgr.14254) 548 : audit [DBG] from='client.14534 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:13:54.841 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:54 smithi132 bash[18607]: audit 2024-04-25T20:13:53.143552+0000 mgr.x (mgr.14254) 548 : audit [DBG] from='client.14534 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:13:56.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:55 smithi132 bash[18607]: cluster 2024-04-25T20:13:54.520183+0000 mgr.x (mgr.14254) 549 : cluster [DBG] pgmap v447: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:56.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:55 smithi132 bash[18607]: cluster 2024-04-25T20:13:54.520183+0000 mgr.x (mgr.14254) 549 : cluster [DBG] pgmap v447: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:58.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:57 smithi132 bash[18607]: cluster 2024-04-25T20:13:56.521630+0000 mgr.x (mgr.14254) 550 : cluster [DBG] pgmap v448: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:58.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:57 smithi132 bash[18607]: cluster 2024-04-25T20:13:56.521630+0000 mgr.x (mgr.14254) 550 : cluster [DBG] pgmap v448: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:13:59.587 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:14:00.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:59 smithi132 bash[18607]: cluster 2024-04-25T20:13:58.523468+0000 mgr.x (mgr.14254) 551 : cluster [DBG] pgmap v449: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:00.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:13:59 smithi132 bash[18607]: cluster 2024-04-25T20:13:58.523468+0000 mgr.x (mgr.14254) 551 : cluster [DBG] pgmap v449: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:01.897 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:01 smithi132 bash[18607]: cluster 2024-04-25T20:14:00.524183+0000 mgr.x (mgr.14254) 552 : cluster [DBG] pgmap v450: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:01.897 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:01 smithi132 bash[18607]: cluster 2024-04-25T20:14:00.524183+0000 mgr.x (mgr.14254) 552 : cluster [DBG] pgmap v450: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:02.294 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:14:02.295 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:14:02.997 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:02 smithi132 bash[18607]: audit 2024-04-25T20:14:02.301187+0000 mgr.x (mgr.14254) 553 : audit [DBG] from='client.14536 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:14:02.997 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:02 smithi132 bash[18607]: audit 2024-04-25T20:14:02.301187+0000 mgr.x (mgr.14254) 553 : audit [DBG] from='client.14536 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:14:02.997 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:02 smithi132 bash[18607]: cluster 2024-04-25T20:14:02.525712+0000 mgr.x (mgr.14254) 554 : cluster [DBG] pgmap v451: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:02.998 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:02 smithi132 bash[18607]: cluster 2024-04-25T20:14:02.525712+0000 mgr.x (mgr.14254) 554 : cluster [DBG] pgmap v451: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:02.999 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:14:04.000 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:14:06.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:05 smithi132 bash[18607]: cluster 2024-04-25T20:14:04.527807+0000 mgr.x (mgr.14254) 555 : cluster [DBG] pgmap v452: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:06.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:05 smithi132 bash[18607]: cluster 2024-04-25T20:14:04.527807+0000 mgr.x (mgr.14254) 555 : cluster [DBG] pgmap v452: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:08.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:07 smithi132 bash[18607]: cluster 2024-04-25T20:14:06.529446+0000 mgr.x (mgr.14254) 556 : cluster [DBG] pgmap v453: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:08.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:07 smithi132 bash[18607]: cluster 2024-04-25T20:14:06.529446+0000 mgr.x (mgr.14254) 556 : cluster [DBG] pgmap v453: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:08.757 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:14:09.840 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:09 smithi132 bash[18607]: cluster 2024-04-25T20:14:08.531510+0000 mgr.x (mgr.14254) 557 : cluster [DBG] pgmap v454: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:09.840 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:09 smithi132 bash[18607]: cluster 2024-04-25T20:14:08.531510+0000 mgr.x (mgr.14254) 557 : cluster [DBG] pgmap v454: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:10.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:10 smithi132 bash[18607]: cluster 2024-04-25T20:14:10.532450+0000 mgr.x (mgr.14254) 558 : cluster [DBG] pgmap v455: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:10.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:10 smithi132 bash[18607]: cluster 2024-04-25T20:14:10.532450+0000 mgr.x (mgr.14254) 558 : cluster [DBG] pgmap v455: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:11.591 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:14:11.592 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:14:12.218 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:14:13.219 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:14:14.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:13 smithi132 bash[18607]: audit 2024-04-25T20:14:11.598296+0000 mgr.x (mgr.14254) 559 : audit [DBG] from='client.14538 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:14:14.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:13 smithi132 bash[18607]: audit 2024-04-25T20:14:11.598296+0000 mgr.x (mgr.14254) 559 : audit [DBG] from='client.14538 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:14:14.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:13 smithi132 bash[18607]: cluster 2024-04-25T20:14:12.533808+0000 mgr.x (mgr.14254) 560 : cluster [DBG] pgmap v456: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:14.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:13 smithi132 bash[18607]: cluster 2024-04-25T20:14:12.533808+0000 mgr.x (mgr.14254) 560 : cluster [DBG] pgmap v456: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:16.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:15 smithi132 bash[18607]: cluster 2024-04-25T20:14:14.535589+0000 mgr.x (mgr.14254) 561 : cluster [DBG] pgmap v457: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:16.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:15 smithi132 bash[18607]: cluster 2024-04-25T20:14:14.535589+0000 mgr.x (mgr.14254) 561 : cluster [DBG] pgmap v457: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:17.983 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:14:18.004 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:17 smithi132 bash[18607]: cluster 2024-04-25T20:14:16.536996+0000 mgr.x (mgr.14254) 562 : cluster [DBG] pgmap v458: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:18.004 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:17 smithi132 bash[18607]: cluster 2024-04-25T20:14:16.536996+0000 mgr.x (mgr.14254) 562 : cluster [DBG] pgmap v458: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:20.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:19 smithi132 bash[18607]: cluster 2024-04-25T20:14:18.538794+0000 mgr.x (mgr.14254) 563 : cluster [DBG] pgmap v459: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:20.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:19 smithi132 bash[18607]: cluster 2024-04-25T20:14:18.538794+0000 mgr.x (mgr.14254) 563 : cluster [DBG] pgmap v459: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:20.699 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:14:20.699 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:14:20.911 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:20 smithi132 bash[18607]: cluster 2024-04-25T20:14:20.540189+0000 mgr.x (mgr.14254) 564 : cluster [DBG] pgmap v460: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:20.911 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:20 smithi132 bash[18607]: cluster 2024-04-25T20:14:20.540189+0000 mgr.x (mgr.14254) 564 : cluster [DBG] pgmap v460: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:21.370 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:14:21.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:21 smithi132 bash[18607]: audit 2024-04-25T20:14:20.703975+0000 mgr.x (mgr.14254) 565 : audit [DBG] from='client.14540 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:14:21.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:21 smithi132 bash[18607]: audit 2024-04-25T20:14:20.703975+0000 mgr.x (mgr.14254) 565 : audit [DBG] from='client.14540 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:14:22.371 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:14:22.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:22 smithi132 bash[18607]: cluster 2024-04-25T20:14:22.541633+0000 mgr.x (mgr.14254) 566 : cluster [DBG] pgmap v461: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:22.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:22 smithi132 bash[18607]: cluster 2024-04-25T20:14:22.541633+0000 mgr.x (mgr.14254) 566 : cluster [DBG] pgmap v461: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:25.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:24 smithi132 bash[18607]: cluster 2024-04-25T20:14:24.543428+0000 mgr.x (mgr.14254) 567 : cluster [DBG] pgmap v462: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:25.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:24 smithi132 bash[18607]: cluster 2024-04-25T20:14:24.543428+0000 mgr.x (mgr.14254) 567 : cluster [DBG] pgmap v462: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:27.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:26 smithi132 bash[18607]: cluster 2024-04-25T20:14:26.544953+0000 mgr.x (mgr.14254) 568 : cluster [DBG] pgmap v463: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:27.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:26 smithi132 bash[18607]: cluster 2024-04-25T20:14:26.544953+0000 mgr.x (mgr.14254) 568 : cluster [DBG] pgmap v463: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:27.124 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:14:29.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:28 smithi132 bash[18607]: cluster 2024-04-25T20:14:28.546758+0000 mgr.x (mgr.14254) 569 : cluster [DBG] pgmap v464: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:29.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:28 smithi132 bash[18607]: cluster 2024-04-25T20:14:28.546758+0000 mgr.x (mgr.14254) 569 : cluster [DBG] pgmap v464: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:29.961 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:14:29.961 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:14:30.593 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:14:31.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:30 smithi132 bash[18607]: audit 2024-04-25T20:14:29.967494+0000 mgr.x (mgr.14254) 570 : audit [DBG] from='client.14542 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:14:31.016 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:30 smithi132 bash[18607]: audit 2024-04-25T20:14:29.967494+0000 mgr.x (mgr.14254) 570 : audit [DBG] from='client.14542 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:14:31.016 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:30 smithi132 bash[18607]: cluster 2024-04-25T20:14:30.548239+0000 mgr.x (mgr.14254) 571 : cluster [DBG] pgmap v465: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:31.016 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:30 smithi132 bash[18607]: cluster 2024-04-25T20:14:30.548239+0000 mgr.x (mgr.14254) 571 : cluster [DBG] pgmap v465: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:31.594 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:14:33.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:32 smithi132 bash[18607]: cluster 2024-04-25T20:14:32.549665+0000 mgr.x (mgr.14254) 572 : cluster [DBG] pgmap v466: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:33.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:32 smithi132 bash[18607]: cluster 2024-04-25T20:14:32.549665+0000 mgr.x (mgr.14254) 572 : cluster [DBG] pgmap v466: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:35.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:34 smithi132 bash[18607]: cluster 2024-04-25T20:14:34.551562+0000 mgr.x (mgr.14254) 573 : cluster [DBG] pgmap v467: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:35.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:34 smithi132 bash[18607]: cluster 2024-04-25T20:14:34.551562+0000 mgr.x (mgr.14254) 573 : cluster [DBG] pgmap v467: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:36.346 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:14:37.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:36 smithi132 bash[18607]: cluster 2024-04-25T20:14:36.553209+0000 mgr.x (mgr.14254) 574 : cluster [DBG] pgmap v468: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:37.016 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:36 smithi132 bash[18607]: cluster 2024-04-25T20:14:36.553209+0000 mgr.x (mgr.14254) 574 : cluster [DBG] pgmap v468: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:39.112 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:14:39.112 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T19:59:49.410431Z service:smb.modtest1 [INFO] \"service was created\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:14:39.707 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:14:39.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:39 smithi132 bash[18607]: cluster 2024-04-25T20:14:38.555257+0000 mgr.x (mgr.14254) 575 : cluster [DBG] pgmap v469: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:39.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:39 smithi132 bash[18607]: cluster 2024-04-25T20:14:38.555257+0000 mgr.x (mgr.14254) 575 : cluster [DBG] pgmap v469: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:40.708 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:14:40.765 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:40 smithi132 bash[18607]: audit 2024-04-25T20:14:39.118460+0000 mgr.x (mgr.14254) 576 : audit [DBG] from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:14:40.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:40 smithi132 bash[18607]: audit 2024-04-25T20:14:39.118460+0000 mgr.x (mgr.14254) 576 : audit [DBG] from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:14:41.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:41 smithi132 bash[18607]: cluster 2024-04-25T20:14:40.556778+0000 mgr.x (mgr.14254) 577 : cluster [DBG] pgmap v470: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:41.766 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:41 smithi132 bash[18607]: cluster 2024-04-25T20:14:40.556778+0000 mgr.x (mgr.14254) 577 : cluster [DBG] pgmap v470: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:43.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:42 smithi132 bash[18607]: cluster 2024-04-25T20:14:42.558295+0000 mgr.x (mgr.14254) 578 : cluster [DBG] pgmap v471: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:43.016 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:42 smithi132 bash[18607]: cluster 2024-04-25T20:14:42.558295+0000 mgr.x (mgr.14254) 578 : cluster [DBG] pgmap v471: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:45.016 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:44 smithi132 bash[18607]: cluster 2024-04-25T20:14:44.560297+0000 mgr.x (mgr.14254) 579 : cluster [DBG] pgmap v472: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:45.016 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:44 smithi132 bash[18607]: cluster 2024-04-25T20:14:44.560297+0000 mgr.x (mgr.14254) 579 : cluster [DBG] pgmap v472: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:45.469 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:14:46.265 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:46 smithi132 bash[18607]: audit 2024-04-25T20:14:45.955967+0000 mon.a (mon.0) 443 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "auth rm", "entity": "client.smb.config.modtest1.smithi132.jrsyig"}]: dispatch 2024-04-25T20:14:46.265 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:46 smithi132 bash[18607]: audit 2024-04-25T20:14:45.955967+0000 mon.a (mon.0) 443 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "auth rm", "entity": "client.smb.config.modtest1.smithi132.jrsyig"}]: dispatch 2024-04-25T20:14:46.265 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:46 smithi132 bash[18607]: audit 2024-04-25T20:14:45.958027+0000 mon.a (mon.0) 444 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd='[{"prefix": "auth rm", "entity": "client.smb.config.modtest1.smithi132.jrsyig"}]': finished 2024-04-25T20:14:46.265 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:46 smithi132 bash[18607]: audit 2024-04-25T20:14:45.958027+0000 mon.a (mon.0) 444 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd='[{"prefix": "auth rm", "entity": "client.smb.config.modtest1.smithi132.jrsyig"}]': finished 2024-04-25T20:14:46.265 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:46 smithi132 bash[18607]: audit 2024-04-25T20:14:45.988421+0000 mon.a (mon.0) 445 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T20:14:46.265 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:46 smithi132 bash[18607]: audit 2024-04-25T20:14:45.988421+0000 mon.a (mon.0) 445 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T20:14:46.265 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:14:45 smithi132 bash[18860]: debug 2024-04-25T20:14:45.980+0000 7f9113b3f700 -1 log_channel(cephadm) log [ERR] : Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig 2024-04-25T20:14:46.266 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:14:45 smithi132 bash[18860]: /usr/bin/docker: stdout 2024-04-25T20:14:46.266 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:14:45 smithi132 bash[18860]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig 2024-04-25T20:14:46.266 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:14:45 smithi132 bash[18860]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig 2024-04-25T20:14:46.266 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:14:45 smithi132 bash[18860]: /usr/bin/docker: stdout 2024-04-25T20:14:46.266 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:14:45 smithi132 bash[18860]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig 2024-04-25T20:14:46.266 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:14:45 smithi132 bash[18860]: Deploy daemon smb.modtest1.smithi132.jrsyig ... 2024-04-25T20:14:46.266 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:14:45 smithi132 bash[18860]: Verifying port 0.0.0.0:445 ... 2024-04-25T20:14:46.266 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:14:45 smithi132 bash[18860]: systemctl: timeout after 895 seconds 2024-04-25T20:14:46.266 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:14:45 smithi132 bash[18860]: Non-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig 2024-04-25T20:14:46.266 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:14:45 smithi132 bash[18860]: Traceback (most recent call last): 2024-04-25T20:14:46.266 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:14:45 smithi132 bash[18860]: File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main 2024-04-25T20:14:46.266 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:14:45 smithi132 bash[18860]: return _run_code(code, main_globals, None, 2024-04-25T20:14:46.267 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:14:45 smithi132 bash[18860]: File "/usr/lib/python3.10/runpy.py", line 86, in _run_code 2024-04-25T20:14:46.267 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:14:45 smithi132 bash[18860]: exec(code, run_globals) 2024-04-25T20:14:46.267 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:14:45 smithi132 bash[18860]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 5618, in 2024-04-25T20:14:46.267 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:14:45 smithi132 bash[18860]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 5606, in main 2024-04-25T20:14:46.267 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:14:45 smithi132 bash[18860]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 3096, in command_deploy_from 2024-04-25T20:14:46.267 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:14:45 smithi132 bash[18860]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 3131, in _common_deploy 2024-04-25T20:14:46.267 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:14:45 smithi132 bash[18860]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 3151, in _deploy_daemon_container 2024-04-25T20:14:46.267 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:14:45 smithi132 bash[18860]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 1087, in deploy_daemon 2024-04-25T20:14:46.267 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:14:45 smithi132 bash[18860]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 1223, in deploy_daemon_units 2024-04-25T20:14:46.267 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:14:45 smithi132 bash[18860]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-25T20:14:46.267 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:14:45 smithi132 bash[18860]: RuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig 2024-04-25T20:14:47.266 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: cephadm 2024-04-25T20:14:45.955246+0000 mgr.x (mgr.14254) 580 : cephadm [INF] Removing key for client.smb.config.modtest1.smithi132.jrsyig 2024-04-25T20:14:47.267 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: cephadm 2024-04-25T20:14:45.955246+0000 mgr.x (mgr.14254) 580 : cephadm [INF] Removing key for client.smb.config.modtest1.smithi132.jrsyig 2024-04-25T20:14:47.267 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: cephadm 2024-04-25T20:14:45.981431+0000 mgr.x (mgr.14254) 581 : cephadm [ERR] Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig 2024-04-25T20:14:47.267 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: /usr/bin/docker: stdout 2024-04-25T20:14:47.267 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig 2024-04-25T20:14:47.267 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig 2024-04-25T20:14:47.267 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: /usr/bin/docker: stdout 2024-04-25T20:14:47.267 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig 2024-04-25T20:14:47.267 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: Deploy daemon smb.modtest1.smithi132.jrsyig ... 2024-04-25T20:14:47.267 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: Verifying port 0.0.0.0:445 ... 2024-04-25T20:14:47.267 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: systemctl: timeout after 895 seconds 2024-04-25T20:14:47.267 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: Non-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig 2024-04-25T20:14:47.267 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: Traceback (most recent call last): 2024-04-25T20:14:47.268 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main 2024-04-25T20:14:47.268 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: return _run_code(code, main_globals, None, 2024-04-25T20:14:47.268 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: File "/usr/lib/python3.10/runpy.py", line 86, in _run_code 2024-04-25T20:14:47.268 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: exec(code, run_globals) 2024-04-25T20:14:47.268 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 5618, in 2024-04-25T20:14:47.268 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 5606, in main 2024-04-25T20:14:47.268 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 3096, in command_deploy_from 2024-04-25T20:14:47.268 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 3131, in _common_deploy 2024-04-25T20:14:47.268 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 3151, in _deploy_daemon_container 2024-04-25T20:14:47.268 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 1087, in deploy_daemon 2024-04-25T20:14:47.268 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 1223, in deploy_daemon_units 2024-04-25T20:14:47.268 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-25T20:14:47.269 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: RuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig 2024-04-25T20:14:47.269 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: cephadm 2024-04-25T20:14:45.981431+0000 mgr.x (mgr.14254) 581 : cephadm [ERR] Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig 2024-04-25T20:14:47.269 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: /usr/bin/docker: stdout 2024-04-25T20:14:47.269 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig 2024-04-25T20:14:47.269 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig 2024-04-25T20:14:47.269 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: /usr/bin/docker: stdout 2024-04-25T20:14:47.269 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig 2024-04-25T20:14:47.269 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: Deploy daemon smb.modtest1.smithi132.jrsyig ... 2024-04-25T20:14:47.269 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: Verifying port 0.0.0.0:445 ... 2024-04-25T20:14:47.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: systemctl: timeout after 895 seconds 2024-04-25T20:14:47.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: Non-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig 2024-04-25T20:14:47.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: Traceback (most recent call last): 2024-04-25T20:14:47.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main 2024-04-25T20:14:47.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: return _run_code(code, main_globals, None, 2024-04-25T20:14:47.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: File "/usr/lib/python3.10/runpy.py", line 86, in _run_code 2024-04-25T20:14:47.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: exec(code, run_globals) 2024-04-25T20:14:47.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 5618, in 2024-04-25T20:14:47.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 5606, in main 2024-04-25T20:14:47.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 3096, in command_deploy_from 2024-04-25T20:14:47.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 3131, in _common_deploy 2024-04-25T20:14:47.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 3151, in _deploy_daemon_container 2024-04-25T20:14:47.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 1087, in deploy_daemon 2024-04-25T20:14:47.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 1223, in deploy_daemon_units 2024-04-25T20:14:47.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-25T20:14:47.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: RuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig 2024-04-25T20:14:47.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: cephadm 2024-04-25T20:14:45.982951+0000 mgr.x (mgr.14254) 582 : cephadm [WRN] config is a no-op 2024-04-25T20:14:47.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: cephadm 2024-04-25T20:14:45.982951+0000 mgr.x (mgr.14254) 582 : cephadm [WRN] config is a no-op 2024-04-25T20:14:47.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: cluster 2024-04-25T20:14:45.984427+0000 mgr.x (mgr.14254) 583 : cluster [DBG] pgmap v473: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:47.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: cluster 2024-04-25T20:14:45.984427+0000 mgr.x (mgr.14254) 583 : cluster [DBG] pgmap v473: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:47.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: cluster 2024-04-25T20:14:46.957989+0000 mon.a (mon.0) 446 : cluster [WRN] Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2024-04-25T20:14:47.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:47 smithi132 bash[18607]: cluster 2024-04-25T20:14:46.957989+0000 mon.a (mon.0) 446 : cluster [WRN] Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2024-04-25T20:14:48.201 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:14:48.201 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:14:48.390 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:48 smithi132 bash[18607]: audit 2024-04-25T20:14:47.096340+0000 mon.a (mon.0) 447 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T20:14:48.390 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:48 smithi132 bash[18607]: audit 2024-04-25T20:14:47.096340+0000 mon.a (mon.0) 447 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T20:14:48.823 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:14:49.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:49 smithi132 bash[18607]: cluster 2024-04-25T20:14:47.985615+0000 mgr.x (mgr.14254) 584 : cluster [DBG] pgmap v474: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:49.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:49 smithi132 bash[18607]: cluster 2024-04-25T20:14:47.985615+0000 mgr.x (mgr.14254) 584 : cluster [DBG] pgmap v474: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:49.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:49 smithi132 bash[18607]: audit 2024-04-25T20:14:48.207756+0000 mgr.x (mgr.14254) 585 : audit [DBG] from='client.14546 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:14:49.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:49 smithi132 bash[18607]: audit 2024-04-25T20:14:48.207756+0000 mgr.x (mgr.14254) 585 : audit [DBG] from='client.14546 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:14:49.824 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:14:51.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:51 smithi132 bash[18607]: cluster 2024-04-25T20:14:49.987395+0000 mgr.x (mgr.14254) 586 : cluster [DBG] pgmap v475: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:51.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:51 smithi132 bash[18607]: cluster 2024-04-25T20:14:49.987395+0000 mgr.x (mgr.14254) 586 : cluster [DBG] pgmap v475: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:53.415 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:53 smithi132 bash[18607]: cluster 2024-04-25T20:14:51.988995+0000 mgr.x (mgr.14254) 587 : cluster [DBG] pgmap v476: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:53.415 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:53 smithi132 bash[18607]: cluster 2024-04-25T20:14:51.988995+0000 mgr.x (mgr.14254) 587 : cluster [DBG] pgmap v476: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:54.583 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:14:55.266 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:55 smithi132 bash[18607]: cluster 2024-04-25T20:14:53.989996+0000 mgr.x (mgr.14254) 588 : cluster [DBG] pgmap v477: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:55.266 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:55 smithi132 bash[18607]: cluster 2024-04-25T20:14:53.989996+0000 mgr.x (mgr.14254) 588 : cluster [DBG] pgmap v477: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:57.414 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:57 smithi132 bash[18607]: cluster 2024-04-25T20:14:55.990765+0000 mgr.x (mgr.14254) 589 : cluster [DBG] pgmap v478: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:57.414 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:57 smithi132 bash[18607]: cluster 2024-04-25T20:14:55.990765+0000 mgr.x (mgr.14254) 589 : cluster [DBG] pgmap v478: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:58.313 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:14:58.313 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:14:59.035 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:14:59.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:59 smithi132 bash[18607]: cluster 2024-04-25T20:14:57.991665+0000 mgr.x (mgr.14254) 590 : cluster [DBG] pgmap v479: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:59.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:59 smithi132 bash[18607]: cluster 2024-04-25T20:14:57.991665+0000 mgr.x (mgr.14254) 590 : cluster [DBG] pgmap v479: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:14:59.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:59 smithi132 bash[18607]: audit 2024-04-25T20:14:58.317098+0000 mgr.x (mgr.14254) 591 : audit [DBG] from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:14:59.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:14:59 smithi132 bash[18607]: audit 2024-04-25T20:14:58.317098+0000 mgr.x (mgr.14254) 591 : audit [DBG] from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:15:00.036 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:15:01.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:01 smithi132 bash[18607]: cluster 2024-04-25T20:14:59.993558+0000 mgr.x (mgr.14254) 592 : cluster [DBG] pgmap v480: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:01.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:01 smithi132 bash[18607]: cluster 2024-04-25T20:14:59.993558+0000 mgr.x (mgr.14254) 592 : cluster [DBG] pgmap v480: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:03.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:03 smithi132 bash[18607]: cluster 2024-04-25T20:15:01.995231+0000 mgr.x (mgr.14254) 593 : cluster [DBG] pgmap v481: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:03.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:03 smithi132 bash[18607]: cluster 2024-04-25T20:15:01.995231+0000 mgr.x (mgr.14254) 593 : cluster [DBG] pgmap v481: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:04.787 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:15:05.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:05 smithi132 bash[18607]: cluster 2024-04-25T20:15:03.997318+0000 mgr.x (mgr.14254) 594 : cluster [DBG] pgmap v482: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:05.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:05 smithi132 bash[18607]: cluster 2024-04-25T20:15:03.997318+0000 mgr.x (mgr.14254) 594 : cluster [DBG] pgmap v482: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:07.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:07 smithi132 bash[18607]: cluster 2024-04-25T20:15:05.998083+0000 mgr.x (mgr.14254) 595 : cluster [DBG] pgmap v483: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:07.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:07 smithi132 bash[18607]: cluster 2024-04-25T20:15:05.998083+0000 mgr.x (mgr.14254) 595 : cluster [DBG] pgmap v483: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:07.540 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:15:07.540 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:15:08.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:08 smithi132 bash[18607]: audit 2024-04-25T20:15:07.545920+0000 mgr.x (mgr.14254) 596 : audit [DBG] from='client.14552 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:15:08.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:08 smithi132 bash[18607]: audit 2024-04-25T20:15:07.545920+0000 mgr.x (mgr.14254) 596 : audit [DBG] from='client.14552 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:15:08.671 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:15:09.352 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:09 smithi132 bash[18607]: cluster 2024-04-25T20:15:07.999814+0000 mgr.x (mgr.14254) 597 : cluster [DBG] pgmap v484: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:09.352 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:09 smithi132 bash[18607]: cluster 2024-04-25T20:15:07.999814+0000 mgr.x (mgr.14254) 597 : cluster [DBG] pgmap v484: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:09.672 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:15:11.266 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:11 smithi132 bash[18607]: cluster 2024-04-25T20:15:10.000741+0000 mgr.x (mgr.14254) 598 : cluster [DBG] pgmap v485: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:11.266 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:11 smithi132 bash[18607]: cluster 2024-04-25T20:15:10.000741+0000 mgr.x (mgr.14254) 598 : cluster [DBG] pgmap v485: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:13.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:13 smithi132 bash[18607]: cluster 2024-04-25T20:15:12.001657+0000 mgr.x (mgr.14254) 599 : cluster [DBG] pgmap v486: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:13.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:13 smithi132 bash[18607]: cluster 2024-04-25T20:15:12.001657+0000 mgr.x (mgr.14254) 599 : cluster [DBG] pgmap v486: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:15.166 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:15:15.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:15 smithi132 bash[18607]: cluster 2024-04-25T20:15:14.003631+0000 mgr.x (mgr.14254) 600 : cluster [DBG] pgmap v487: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:15.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:15 smithi132 bash[18607]: cluster 2024-04-25T20:15:14.003631+0000 mgr.x (mgr.14254) 600 : cluster [DBG] pgmap v487: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:17.419 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:17 smithi132 bash[18607]: cluster 2024-04-25T20:15:16.005386+0000 mgr.x (mgr.14254) 601 : cluster [DBG] pgmap v488: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:17.419 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:17 smithi132 bash[18607]: cluster 2024-04-25T20:15:16.005386+0000 mgr.x (mgr.14254) 601 : cluster [DBG] pgmap v488: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:17.823 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:15:17.823 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:15:18.485 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:15:19.486 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:15:19.497 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:19 smithi132 bash[18607]: audit 2024-04-25T20:15:17.826539+0000 mgr.x (mgr.14254) 602 : audit [DBG] from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:15:19.497 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:19 smithi132 bash[18607]: audit 2024-04-25T20:15:17.826539+0000 mgr.x (mgr.14254) 602 : audit [DBG] from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:15:19.497 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:19 smithi132 bash[18607]: cluster 2024-04-25T20:15:18.007879+0000 mgr.x (mgr.14254) 603 : cluster [DBG] pgmap v489: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:19.498 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:19 smithi132 bash[18607]: cluster 2024-04-25T20:15:18.007879+0000 mgr.x (mgr.14254) 603 : cluster [DBG] pgmap v489: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:21.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:21 smithi132 bash[18607]: cluster 2024-04-25T20:15:20.009712+0000 mgr.x (mgr.14254) 604 : cluster [DBG] pgmap v490: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:21.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:21 smithi132 bash[18607]: cluster 2024-04-25T20:15:20.009712+0000 mgr.x (mgr.14254) 604 : cluster [DBG] pgmap v490: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:23.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:23 smithi132 bash[18607]: cluster 2024-04-25T20:15:22.011334+0000 mgr.x (mgr.14254) 605 : cluster [DBG] pgmap v491: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:23.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:23 smithi132 bash[18607]: cluster 2024-04-25T20:15:22.011334+0000 mgr.x (mgr.14254) 605 : cluster [DBG] pgmap v491: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:24.236 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:15:25.375 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:25 smithi132 bash[18607]: cluster 2024-04-25T20:15:24.013380+0000 mgr.x (mgr.14254) 606 : cluster [DBG] pgmap v492: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:25.375 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:25 smithi132 bash[18607]: cluster 2024-04-25T20:15:24.013380+0000 mgr.x (mgr.14254) 606 : cluster [DBG] pgmap v492: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:27.193 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:27 smithi132 bash[18607]: cluster 2024-04-25T20:15:26.014204+0000 mgr.x (mgr.14254) 607 : cluster [DBG] pgmap v493: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:27.194 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:27 smithi132 bash[18607]: cluster 2024-04-25T20:15:26.014204+0000 mgr.x (mgr.14254) 607 : cluster [DBG] pgmap v493: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:27.930 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:15:27.930 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:15:28.682 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:15:29.437 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:29 smithi132 bash[18607]: audit 2024-04-25T20:15:27.936013+0000 mgr.x (mgr.14254) 608 : audit [DBG] from='client.14559 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:15:29.437 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:29 smithi132 bash[18607]: audit 2024-04-25T20:15:27.936013+0000 mgr.x (mgr.14254) 608 : audit [DBG] from='client.14559 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:15:29.437 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:29 smithi132 bash[18607]: cluster 2024-04-25T20:15:28.015067+0000 mgr.x (mgr.14254) 609 : cluster [DBG] pgmap v494: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:29.437 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:29 smithi132 bash[18607]: cluster 2024-04-25T20:15:28.015067+0000 mgr.x (mgr.14254) 609 : cluster [DBG] pgmap v494: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:29.683 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:15:31.516 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:31 smithi132 bash[18607]: cluster 2024-04-25T20:15:30.016857+0000 mgr.x (mgr.14254) 610 : cluster [DBG] pgmap v495: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:31.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:31 smithi132 bash[18607]: cluster 2024-04-25T20:15:30.016857+0000 mgr.x (mgr.14254) 610 : cluster [DBG] pgmap v495: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:33.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:33 smithi132 bash[18607]: cluster 2024-04-25T20:15:32.018470+0000 mgr.x (mgr.14254) 611 : cluster [DBG] pgmap v496: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:33.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:33 smithi132 bash[18607]: cluster 2024-04-25T20:15:32.018470+0000 mgr.x (mgr.14254) 611 : cluster [DBG] pgmap v496: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:34.434 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:15:35.390 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:35 smithi132 bash[18607]: cluster 2024-04-25T20:15:34.021153+0000 mgr.x (mgr.14254) 612 : cluster [DBG] pgmap v497: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:35.390 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:35 smithi132 bash[18607]: cluster 2024-04-25T20:15:34.021153+0000 mgr.x (mgr.14254) 612 : cluster [DBG] pgmap v497: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:37.156 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:15:37.157 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:15:37.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:37 smithi132 bash[18607]: cluster 2024-04-25T20:15:36.022231+0000 mgr.x (mgr.14254) 613 : cluster [DBG] pgmap v498: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:37.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:37 smithi132 bash[18607]: cluster 2024-04-25T20:15:36.022231+0000 mgr.x (mgr.14254) 613 : cluster [DBG] pgmap v498: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:37.850 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:15:38.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:38 smithi132 bash[18607]: audit 2024-04-25T20:15:37.162233+0000 mgr.x (mgr.14254) 614 : audit [DBG] from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:15:38.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:38 smithi132 bash[18607]: audit 2024-04-25T20:15:37.162233+0000 mgr.x (mgr.14254) 614 : audit [DBG] from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:15:38.851 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:15:39.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:39 smithi132 bash[18607]: cluster 2024-04-25T20:15:38.024104+0000 mgr.x (mgr.14254) 615 : cluster [DBG] pgmap v499: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:39.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:39 smithi132 bash[18607]: cluster 2024-04-25T20:15:38.024104+0000 mgr.x (mgr.14254) 615 : cluster [DBG] pgmap v499: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:41.267 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:41 smithi132 bash[18607]: cluster 2024-04-25T20:15:40.025028+0000 mgr.x (mgr.14254) 616 : cluster [DBG] pgmap v500: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:41.267 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:41 smithi132 bash[18607]: cluster 2024-04-25T20:15:40.025028+0000 mgr.x (mgr.14254) 616 : cluster [DBG] pgmap v500: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:43.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:43 smithi132 bash[18607]: cluster 2024-04-25T20:15:42.026630+0000 mgr.x (mgr.14254) 617 : cluster [DBG] pgmap v501: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:43.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:43 smithi132 bash[18607]: cluster 2024-04-25T20:15:42.026630+0000 mgr.x (mgr.14254) 617 : cluster [DBG] pgmap v501: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:44.997 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:15:45.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:45 smithi132 bash[18607]: cluster 2024-04-25T20:15:44.028810+0000 mgr.x (mgr.14254) 618 : cluster [DBG] pgmap v502: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:45.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:45 smithi132 bash[18607]: cluster 2024-04-25T20:15:44.028810+0000 mgr.x (mgr.14254) 618 : cluster [DBG] pgmap v502: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:46.389 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:46 smithi132 bash[18607]: audit 2024-04-25T20:15:46.024336+0000 mon.a (mon.0) 448 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T20:15:46.389 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:46 smithi132 bash[18607]: audit 2024-04-25T20:15:46.024336+0000 mon.a (mon.0) 448 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T20:15:47.726 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:15:47.726 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T19:59:17.908921Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T19:59:17.908542Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T19:59:17.908396Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T19:59:17.908684Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "running": 0, "size": 1}}] 2024-04-25T20:15:47.740 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:47 smithi132 bash[18607]: cluster 2024-04-25T20:15:46.030365+0000 mgr.x (mgr.14254) 619 : cluster [DBG] pgmap v503: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:47.740 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:47 smithi132 bash[18607]: cluster 2024-04-25T20:15:46.030365+0000 mgr.x (mgr.14254) 619 : cluster [DBG] pgmap v503: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:47.740 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:47 smithi132 bash[18607]: audit 2024-04-25T20:15:46.500530+0000 mon.a (mon.0) 449 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T20:15:47.740 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:47 smithi132 bash[18607]: audit 2024-04-25T20:15:46.500530+0000 mon.a (mon.0) 449 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T20:15:47.740 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:47 smithi132 bash[18607]: audit 2024-04-25T20:15:46.510405+0000 mon.a (mon.0) 450 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T20:15:47.740 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:47 smithi132 bash[18607]: audit 2024-04-25T20:15:46.510405+0000 mon.a (mon.0) 450 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T20:15:48.391 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:15:49.393 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:15:49.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:49 smithi132 bash[18607]: audit 2024-04-25T20:15:47.728579+0000 mgr.x (mgr.14254) 620 : audit [DBG] from='client.14566 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:15:49.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:49 smithi132 bash[18607]: audit 2024-04-25T20:15:47.728579+0000 mgr.x (mgr.14254) 620 : audit [DBG] from='client.14566 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:15:49.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:49 smithi132 bash[18607]: cluster 2024-04-25T20:15:48.032219+0000 mgr.x (mgr.14254) 621 : cluster [DBG] pgmap v504: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:49.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:49 smithi132 bash[18607]: cluster 2024-04-25T20:15:48.032219+0000 mgr.x (mgr.14254) 621 : cluster [DBG] pgmap v504: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:51.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:51 smithi132 bash[18607]: cluster 2024-04-25T20:15:50.034063+0000 mgr.x (mgr.14254) 622 : cluster [DBG] pgmap v505: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:51.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:51 smithi132 bash[18607]: cluster 2024-04-25T20:15:50.034063+0000 mgr.x (mgr.14254) 622 : cluster [DBG] pgmap v505: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:53.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:53 smithi132 bash[18607]: cluster 2024-04-25T20:15:52.035277+0000 mgr.x (mgr.14254) 623 : cluster [DBG] pgmap v506: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:53.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:53 smithi132 bash[18607]: cluster 2024-04-25T20:15:52.035277+0000 mgr.x (mgr.14254) 623 : cluster [DBG] pgmap v506: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:53.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:53 smithi132 bash[18607]: audit 2024-04-25T20:15:52.314524+0000 mon.a (mon.0) 451 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T20:15:53.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:53 smithi132 bash[18607]: audit 2024-04-25T20:15:52.314524+0000 mon.a (mon.0) 451 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T20:15:53.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:53 smithi132 bash[18607]: audit 2024-04-25T20:15:52.322292+0000 mon.a (mon.0) 452 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T20:15:53.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:53 smithi132 bash[18607]: audit 2024-04-25T20:15:52.322292+0000 mon.a (mon.0) 452 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T20:15:53.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:53 smithi132 bash[18607]: audit 2024-04-25T20:15:52.946174+0000 mon.a (mon.0) 453 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T20:15:53.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:53 smithi132 bash[18607]: audit 2024-04-25T20:15:52.946174+0000 mon.a (mon.0) 453 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T20:15:53.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:53 smithi132 bash[18607]: audit 2024-04-25T20:15:52.947799+0000 mon.a (mon.0) 454 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T20:15:53.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:53 smithi132 bash[18607]: audit 2024-04-25T20:15:52.947799+0000 mon.a (mon.0) 454 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T20:15:53.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:53 smithi132 bash[18607]: audit 2024-04-25T20:15:52.953888+0000 mon.a (mon.0) 455 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T20:15:53.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:53 smithi132 bash[18607]: audit 2024-04-25T20:15:52.953888+0000 mon.a (mon.0) 455 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T20:15:53.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:53 smithi132 bash[18607]: audit 2024-04-25T20:15:52.957816+0000 mon.a (mon.0) 456 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T20:15:53.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:53 smithi132 bash[18607]: audit 2024-04-25T20:15:52.957816+0000 mon.a (mon.0) 456 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T20:15:53.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:53 smithi132 bash[18607]: audit 2024-04-25T20:15:52.959447+0000 mon.a (mon.0) 457 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T20:15:53.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:53 smithi132 bash[18607]: audit 2024-04-25T20:15:52.959447+0000 mon.a (mon.0) 457 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T20:15:53.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:53 smithi132 bash[18607]: audit 2024-04-25T20:15:52.970198+0000 mon.a (mon.0) 458 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T20:15:53.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:53 smithi132 bash[18607]: audit 2024-04-25T20:15:52.970198+0000 mon.a (mon.0) 458 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T20:15:53.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:53 smithi132 bash[18607]: audit 2024-04-25T20:15:53.010464+0000 mon.a (mon.0) 459 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.smb.config.modtest1.smithi132.jrsyig", "caps": ["mon", "allow r, allow command \"config-key get\" with \"key\" prefix \"smb/config/modtest1/\"", "osd", "allow r pool=.smb"]}]: dispatch 2024-04-25T20:15:53.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:53 smithi132 bash[18607]: audit 2024-04-25T20:15:53.010464+0000 mon.a (mon.0) 459 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.smb.config.modtest1.smithi132.jrsyig", "caps": ["mon", "allow r, allow command \"config-key get\" with \"key\" prefix \"smb/config/modtest1/\"", "osd", "allow r pool=.smb"]}]: dispatch 2024-04-25T20:15:53.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:53 smithi132 bash[18607]: audit 2024-04-25T20:15:53.014492+0000 mon.a (mon.0) 460 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "client.smb.config.modtest1.smithi132.jrsyig", "caps": ["mon", "allow r, allow command \"config-key get\" with \"key\" prefix \"smb/config/modtest1/\"", "osd", "allow r pool=.smb"]}]': finished 2024-04-25T20:15:53.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:53 smithi132 bash[18607]: audit 2024-04-25T20:15:53.014492+0000 mon.a (mon.0) 460 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "client.smb.config.modtest1.smithi132.jrsyig", "caps": ["mon", "allow r, allow command \"config-key get\" with \"key\" prefix \"smb/config/modtest1/\"", "osd", "allow r pool=.smb"]}]': finished 2024-04-25T20:15:53.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:53 smithi132 bash[18607]: audit 2024-04-25T20:15:53.019478+0000 mon.a (mon.0) 461 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.smb.fs.cluster.modtest1"}]: dispatch 2024-04-25T20:15:53.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:53 smithi132 bash[18607]: audit 2024-04-25T20:15:53.019478+0000 mon.a (mon.0) 461 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.smb.fs.cluster.modtest1"}]: dispatch 2024-04-25T20:15:53.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:53 smithi132 bash[18607]: audit 2024-04-25T20:15:53.020853+0000 mon.a (mon.0) 462 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T20:15:53.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:53 smithi132 bash[18607]: audit 2024-04-25T20:15:53.020853+0000 mon.a (mon.0) 462 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T20:15:53.924 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:15:54.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:54 smithi132 bash[18607]: cephadm 2024-04-25T20:15:52.948899+0000 mgr.x (mgr.14254) 624 : cephadm [INF] Adjusting osd_memory_target on smithi132 to 4024M 2024-04-25T20:15:54.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:54 smithi132 bash[18607]: cephadm 2024-04-25T20:15:52.948899+0000 mgr.x (mgr.14254) 624 : cephadm [INF] Adjusting osd_memory_target on smithi132 to 4024M 2024-04-25T20:15:54.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:54 smithi132 bash[18607]: cluster 2024-04-25T20:15:52.963622+0000 mgr.x (mgr.14254) 625 : cluster [DBG] pgmap v507: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:54.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:54 smithi132 bash[18607]: cluster 2024-04-25T20:15:52.963622+0000 mgr.x (mgr.14254) 625 : cluster [DBG] pgmap v507: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:54.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:54 smithi132 bash[18607]: cluster 2024-04-25T20:15:52.972126+0000 mgr.x (mgr.14254) 626 : cluster [DBG] pgmap v508: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:54.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:54 smithi132 bash[18607]: cluster 2024-04-25T20:15:52.972126+0000 mgr.x (mgr.14254) 626 : cluster [DBG] pgmap v508: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:54.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:54 smithi132 bash[18607]: cephadm 2024-04-25T20:15:53.009470+0000 mgr.x (mgr.14254) 627 : cephadm [INF] Reconfiguring smb.modtest1.smithi132.jrsyig (unknown last config time)... 2024-04-25T20:15:54.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:54 smithi132 bash[18607]: cephadm 2024-04-25T20:15:53.009470+0000 mgr.x (mgr.14254) 627 : cephadm [INF] Reconfiguring smb.modtest1.smithi132.jrsyig (unknown last config time)... 2024-04-25T20:15:54.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:54 smithi132 bash[18607]: cephadm 2024-04-25T20:15:53.022816+0000 mgr.x (mgr.14254) 628 : cephadm [INF] Reconfiguring daemon smb.modtest1.smithi132.jrsyig on smithi132 2024-04-25T20:15:54.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:54 smithi132 bash[18607]: cephadm 2024-04-25T20:15:53.022816+0000 mgr.x (mgr.14254) 628 : cephadm [INF] Reconfiguring daemon smb.modtest1.smithi132.jrsyig on smithi132 2024-04-25T20:15:54.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:54 smithi132 bash[18607]: cluster 2024-04-25T20:15:53.950519+0000 mon.a (mon.0) 463 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-25T20:15:54.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:54 smithi132 bash[18607]: cluster 2024-04-25T20:15:53.950519+0000 mon.a (mon.0) 463 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-25T20:15:54.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:54 smithi132 bash[18607]: cluster 2024-04-25T20:15:53.950563+0000 mon.a (mon.0) 464 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 1 daemon(s)) 2024-04-25T20:15:54.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:54 smithi132 bash[18607]: cluster 2024-04-25T20:15:53.950563+0000 mon.a (mon.0) 464 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 1 daemon(s)) 2024-04-25T20:15:56.628 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:56 smithi132 bash[18607]: cluster 2024-04-25T20:15:54.973765+0000 mgr.x (mgr.14254) 629 : cluster [DBG] pgmap v509: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:56.628 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:56 smithi132 bash[18607]: cluster 2024-04-25T20:15:54.973765+0000 mgr.x (mgr.14254) 629 : cluster [DBG] pgmap v509: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:58.395 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:15:58.396 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:15:58.619 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:58 smithi132 bash[18607]: cluster 2024-04-25T20:15:56.974667+0000 mgr.x (mgr.14254) 630 : cluster [DBG] pgmap v510: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:58.619 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:58 smithi132 bash[18607]: cluster 2024-04-25T20:15:56.974667+0000 mgr.x (mgr.14254) 630 : cluster [DBG] pgmap v510: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:15:59.156 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:15:59.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:59 smithi132 bash[18607]: audit 2024-04-25T20:15:58.401138+0000 mgr.x (mgr.14254) 631 : audit [DBG] from='client.14570 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:15:59.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:59 smithi132 bash[18607]: audit 2024-04-25T20:15:58.401138+0000 mgr.x (mgr.14254) 631 : audit [DBG] from='client.14570 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:15:59.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:59 smithi132 bash[18607]: audit 2024-04-25T20:15:58.633383+0000 mon.a (mon.0) 465 : audit [DBG] from='client.? 172.17.0.2:0/1024310442' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:15:59.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:15:59 smithi132 bash[18607]: audit 2024-04-25T20:15:58.633383+0000 mon.a (mon.0) 465 : audit [DBG] from='client.? 172.17.0.2:0/1024310442' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:16:00.158 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:16:00.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:00 smithi132 bash[18607]: cluster 2024-04-25T20:15:58.976517+0000 mgr.x (mgr.14254) 632 : cluster [DBG] pgmap v511: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 229 B/s rd, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:16:00.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:00 smithi132 bash[18607]: cluster 2024-04-25T20:15:58.976517+0000 mgr.x (mgr.14254) 632 : cluster [DBG] pgmap v511: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 229 B/s rd, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:16:02.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:02 smithi132 bash[18607]: cluster 2024-04-25T20:16:00.977910+0000 mgr.x (mgr.14254) 633 : cluster [DBG] pgmap v512: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 229 B/s rd, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:16:02.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:02 smithi132 bash[18607]: cluster 2024-04-25T20:16:00.977910+0000 mgr.x (mgr.14254) 633 : cluster [DBG] pgmap v512: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 229 B/s rd, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:16:04.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:04 smithi132 bash[18607]: cluster 2024-04-25T20:16:02.979612+0000 mgr.x (mgr.14254) 634 : cluster [DBG] pgmap v513: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 204 B/s rd, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:16:04.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:04 smithi132 bash[18607]: cluster 2024-04-25T20:16:02.979612+0000 mgr.x (mgr.14254) 634 : cluster [DBG] pgmap v513: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 204 B/s rd, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:16:04.914 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:16:06.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:06 smithi132 bash[18607]: cluster 2024-04-25T20:16:04.981002+0000 mgr.x (mgr.14254) 635 : cluster [DBG] pgmap v514: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 170 B/s rd, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:16:06.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:06 smithi132 bash[18607]: cluster 2024-04-25T20:16:04.981002+0000 mgr.x (mgr.14254) 635 : cluster [DBG] pgmap v514: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 170 B/s rd, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:16:07.796 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:16:07.797 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:16:08.452 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:16:08.685 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:08 smithi132 bash[18607]: cluster 2024-04-25T20:16:06.982401+0000 mgr.x (mgr.14254) 636 : cluster [DBG] pgmap v515: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 170 B/s rd, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:16:08.685 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:08 smithi132 bash[18607]: cluster 2024-04-25T20:16:06.982401+0000 mgr.x (mgr.14254) 636 : cluster [DBG] pgmap v515: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 170 B/s rd, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:16:09.453 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:16:09.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:09 smithi132 bash[18607]: audit 2024-04-25T20:16:07.798622+0000 mgr.x (mgr.14254) 637 : audit [DBG] from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:16:09.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:09 smithi132 bash[18607]: audit 2024-04-25T20:16:07.798622+0000 mgr.x (mgr.14254) 637 : audit [DBG] from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:16:10.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:10 smithi132 bash[18607]: cluster 2024-04-25T20:16:08.984252+0000 mgr.x (mgr.14254) 638 : cluster [DBG] pgmap v516: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 170 B/s rd, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:16:10.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:10 smithi132 bash[18607]: cluster 2024-04-25T20:16:08.984252+0000 mgr.x (mgr.14254) 638 : cluster [DBG] pgmap v516: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 170 B/s rd, 0 op/s; 27/81 objects degraded (33.333%) 2024-04-25T20:16:10.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:10 smithi132 bash[18607]: audit 2024-04-25T20:16:09.959500+0000 mon.a (mon.0) 466 : audit [DBG] from='client.? 172.17.0.2:0/2729041609' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:16:10.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:10 smithi132 bash[18607]: audit 2024-04-25T20:16:09.959500+0000 mon.a (mon.0) 466 : audit [DBG] from='client.? 172.17.0.2:0/2729041609' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:16:12.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:12 smithi132 bash[18607]: cluster 2024-04-25T20:16:10.985725+0000 mgr.x (mgr.14254) 639 : cluster [DBG] pgmap v517: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:12.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:12 smithi132 bash[18607]: cluster 2024-04-25T20:16:10.985725+0000 mgr.x (mgr.14254) 639 : cluster [DBG] pgmap v517: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:14.217 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:16:14.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:14 smithi132 bash[18607]: cluster 2024-04-25T20:16:12.987329+0000 mgr.x (mgr.14254) 640 : cluster [DBG] pgmap v518: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:14.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:14 smithi132 bash[18607]: cluster 2024-04-25T20:16:12.987329+0000 mgr.x (mgr.14254) 640 : cluster [DBG] pgmap v518: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:16.622 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:16 smithi132 bash[18607]: cluster 2024-04-25T20:16:14.988977+0000 mgr.x (mgr.14254) 641 : cluster [DBG] pgmap v519: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:16.623 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:16 smithi132 bash[18607]: cluster 2024-04-25T20:16:14.988977+0000 mgr.x (mgr.14254) 641 : cluster [DBG] pgmap v519: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:17.005 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:16:17.005 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:16:17.690 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:16:18.691 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:16:18.702 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:18 smithi132 bash[18607]: cluster 2024-04-25T20:16:16.990302+0000 mgr.x (mgr.14254) 642 : cluster [DBG] pgmap v520: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:18.702 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:18 smithi132 bash[18607]: cluster 2024-04-25T20:16:16.990302+0000 mgr.x (mgr.14254) 642 : cluster [DBG] pgmap v520: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:18.702 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:18 smithi132 bash[18607]: audit 2024-04-25T20:16:17.006632+0000 mgr.x (mgr.14254) 643 : audit [DBG] from='client.14578 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:16:18.703 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:18 smithi132 bash[18607]: audit 2024-04-25T20:16:17.006632+0000 mgr.x (mgr.14254) 643 : audit [DBG] from='client.14578 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:16:20.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:20 smithi132 bash[18607]: cluster 2024-04-25T20:16:18.992168+0000 mgr.x (mgr.14254) 644 : cluster [DBG] pgmap v521: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:20.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:20 smithi132 bash[18607]: cluster 2024-04-25T20:16:18.992168+0000 mgr.x (mgr.14254) 644 : cluster [DBG] pgmap v521: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:21.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:21 smithi132 bash[18607]: audit 2024-04-25T20:16:21.291174+0000 mon.a (mon.0) 467 : audit [DBG] from='client.? 172.17.0.2:0/2355175124' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:16:21.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:21 smithi132 bash[18607]: audit 2024-04-25T20:16:21.291174+0000 mon.a (mon.0) 467 : audit [DBG] from='client.? 172.17.0.2:0/2355175124' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:16:22.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:22 smithi132 bash[18607]: cluster 2024-04-25T20:16:20.993690+0000 mgr.x (mgr.14254) 645 : cluster [DBG] pgmap v522: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:22.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:22 smithi132 bash[18607]: cluster 2024-04-25T20:16:20.993690+0000 mgr.x (mgr.14254) 645 : cluster [DBG] pgmap v522: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:23.448 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:16:24.625 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:24 smithi132 bash[18607]: cluster 2024-04-25T20:16:22.995521+0000 mgr.x (mgr.14254) 646 : cluster [DBG] pgmap v523: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:24.625 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:24 smithi132 bash[18607]: cluster 2024-04-25T20:16:22.995521+0000 mgr.x (mgr.14254) 646 : cluster [DBG] pgmap v523: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:26.173 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:16:26.173 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:16:26.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:26 smithi132 bash[18607]: cluster 2024-04-25T20:16:24.997162+0000 mgr.x (mgr.14254) 647 : cluster [DBG] pgmap v524: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:26.517 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:26 smithi132 bash[18607]: cluster 2024-04-25T20:16:24.997162+0000 mgr.x (mgr.14254) 647 : cluster [DBG] pgmap v524: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:26.842 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:16:27.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:27 smithi132 bash[18607]: audit 2024-04-25T20:16:26.175102+0000 mgr.x (mgr.14254) 648 : audit [DBG] from='client.14582 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:16:27.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:27 smithi132 bash[18607]: audit 2024-04-25T20:16:26.175102+0000 mgr.x (mgr.14254) 648 : audit [DBG] from='client.14582 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:16:27.843 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:16:28.767 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:28 smithi132 bash[18607]: cluster 2024-04-25T20:16:26.998548+0000 mgr.x (mgr.14254) 649 : cluster [DBG] pgmap v525: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:28.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:28 smithi132 bash[18607]: cluster 2024-04-25T20:16:26.998548+0000 mgr.x (mgr.14254) 649 : cluster [DBG] pgmap v525: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:30.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:30 smithi132 bash[18607]: cluster 2024-04-25T20:16:29.000368+0000 mgr.x (mgr.14254) 650 : cluster [DBG] pgmap v526: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:30.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:30 smithi132 bash[18607]: cluster 2024-04-25T20:16:29.000368+0000 mgr.x (mgr.14254) 650 : cluster [DBG] pgmap v526: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:32.601 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:16:32.622 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:32 smithi132 bash[18607]: cluster 2024-04-25T20:16:31.001821+0000 mgr.x (mgr.14254) 651 : cluster [DBG] pgmap v527: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:32.622 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:32 smithi132 bash[18607]: cluster 2024-04-25T20:16:31.001821+0000 mgr.x (mgr.14254) 651 : cluster [DBG] pgmap v527: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:33.688 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:33 smithi132 bash[18607]: audit 2024-04-25T20:16:32.617580+0000 mon.a (mon.0) 468 : audit [DBG] from='client.? 172.17.0.2:0/1068253234' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:16:33.689 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:33 smithi132 bash[18607]: audit 2024-04-25T20:16:32.617580+0000 mon.a (mon.0) 468 : audit [DBG] from='client.? 172.17.0.2:0/1068253234' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:16:34.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:34 smithi132 bash[18607]: cluster 2024-04-25T20:16:33.003505+0000 mgr.x (mgr.14254) 652 : cluster [DBG] pgmap v528: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:34.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:34 smithi132 bash[18607]: cluster 2024-04-25T20:16:33.003505+0000 mgr.x (mgr.14254) 652 : cluster [DBG] pgmap v528: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:35.631 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:16:35.631 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:16:36.299 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:16:36.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:36 smithi132 bash[18607]: cluster 2024-04-25T20:16:35.004872+0000 mgr.x (mgr.14254) 653 : cluster [DBG] pgmap v529: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:36.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:36 smithi132 bash[18607]: cluster 2024-04-25T20:16:35.004872+0000 mgr.x (mgr.14254) 653 : cluster [DBG] pgmap v529: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:37.299 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:16:37.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:37 smithi132 bash[18607]: audit 2024-04-25T20:16:35.633235+0000 mgr.x (mgr.14254) 654 : audit [DBG] from='client.14586 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:16:37.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:37 smithi132 bash[18607]: audit 2024-04-25T20:16:35.633235+0000 mgr.x (mgr.14254) 654 : audit [DBG] from='client.14586 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:16:38.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:38 smithi132 bash[18607]: cluster 2024-04-25T20:16:37.006357+0000 mgr.x (mgr.14254) 655 : cluster [DBG] pgmap v530: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:38.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:38 smithi132 bash[18607]: cluster 2024-04-25T20:16:37.006357+0000 mgr.x (mgr.14254) 655 : cluster [DBG] pgmap v530: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:41.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:40 smithi132 bash[18607]: cluster 2024-04-25T20:16:39.008255+0000 mgr.x (mgr.14254) 656 : cluster [DBG] pgmap v531: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:41.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:40 smithi132 bash[18607]: cluster 2024-04-25T20:16:39.008255+0000 mgr.x (mgr.14254) 656 : cluster [DBG] pgmap v531: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:42.057 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:16:42.888 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:42 smithi132 bash[18607]: cluster 2024-04-25T20:16:41.009672+0000 mgr.x (mgr.14254) 657 : cluster [DBG] pgmap v532: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:42.888 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:42 smithi132 bash[18607]: cluster 2024-04-25T20:16:41.009672+0000 mgr.x (mgr.14254) 657 : cluster [DBG] pgmap v532: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:44.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:44 smithi132 bash[18607]: cluster 2024-04-25T20:16:43.010710+0000 mgr.x (mgr.14254) 658 : cluster [DBG] pgmap v533: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:44.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:44 smithi132 bash[18607]: cluster 2024-04-25T20:16:43.010710+0000 mgr.x (mgr.14254) 658 : cluster [DBG] pgmap v533: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:44.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:44 smithi132 bash[18607]: audit 2024-04-25T20:16:43.950369+0000 mon.a (mon.0) 469 : audit [DBG] from='client.? 172.17.0.2:0/808048280' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:16:44.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:44 smithi132 bash[18607]: audit 2024-04-25T20:16:43.950369+0000 mon.a (mon.0) 469 : audit [DBG] from='client.? 172.17.0.2:0/808048280' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:16:44.905 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:16:44.905 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:16:45.651 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:16:46.652 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:16:47.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:46 smithi132 bash[18607]: audit 2024-04-25T20:16:44.909135+0000 mgr.x (mgr.14254) 659 : audit [DBG] from='client.14590 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:16:47.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:46 smithi132 bash[18607]: audit 2024-04-25T20:16:44.909135+0000 mgr.x (mgr.14254) 659 : audit [DBG] from='client.14590 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:16:47.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:46 smithi132 bash[18607]: cluster 2024-04-25T20:16:45.011652+0000 mgr.x (mgr.14254) 660 : cluster [DBG] pgmap v534: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:47.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:46 smithi132 bash[18607]: cluster 2024-04-25T20:16:45.011652+0000 mgr.x (mgr.14254) 660 : cluster [DBG] pgmap v534: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:49.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:48 smithi132 bash[18607]: cluster 2024-04-25T20:16:47.013114+0000 mgr.x (mgr.14254) 661 : cluster [DBG] pgmap v535: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:49.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:48 smithi132 bash[18607]: cluster 2024-04-25T20:16:47.013114+0000 mgr.x (mgr.14254) 661 : cluster [DBG] pgmap v535: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:51.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:50 smithi132 bash[18607]: cluster 2024-04-25T20:16:49.015043+0000 mgr.x (mgr.14254) 662 : cluster [DBG] pgmap v536: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:51.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:50 smithi132 bash[18607]: cluster 2024-04-25T20:16:49.015043+0000 mgr.x (mgr.14254) 662 : cluster [DBG] pgmap v536: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:51.415 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:16:53.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:52 smithi132 bash[18607]: cluster 2024-04-25T20:16:51.016538+0000 mgr.x (mgr.14254) 663 : cluster [DBG] pgmap v537: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:53.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:52 smithi132 bash[18607]: cluster 2024-04-25T20:16:51.016538+0000 mgr.x (mgr.14254) 663 : cluster [DBG] pgmap v537: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:54.204 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:16:54.204 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:16:54.889 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:54 smithi132 bash[18607]: cluster 2024-04-25T20:16:53.017865+0000 mgr.x (mgr.14254) 664 : cluster [DBG] pgmap v538: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:54.889 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:54 smithi132 bash[18607]: cluster 2024-04-25T20:16:53.017865+0000 mgr.x (mgr.14254) 664 : cluster [DBG] pgmap v538: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:54.890 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:16:55.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:55 smithi132 bash[18607]: audit 2024-04-25T20:16:54.205401+0000 mgr.x (mgr.14254) 665 : audit [DBG] from='client.14592 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:16:55.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:55 smithi132 bash[18607]: audit 2024-04-25T20:16:54.205401+0000 mgr.x (mgr.14254) 665 : audit [DBG] from='client.14592 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:16:55.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:55 smithi132 bash[18607]: audit 2024-04-25T20:16:55.282266+0000 mon.a (mon.0) 470 : audit [DBG] from='client.? 172.17.0.2:0/3475111999' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:16:55.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:55 smithi132 bash[18607]: audit 2024-04-25T20:16:55.282266+0000 mon.a (mon.0) 470 : audit [DBG] from='client.? 172.17.0.2:0/3475111999' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:16:55.891 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:16:57.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:56 smithi132 bash[18607]: cluster 2024-04-25T20:16:55.019487+0000 mgr.x (mgr.14254) 666 : cluster [DBG] pgmap v539: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:57.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:56 smithi132 bash[18607]: cluster 2024-04-25T20:16:55.019487+0000 mgr.x (mgr.14254) 666 : cluster [DBG] pgmap v539: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:59.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:58 smithi132 bash[18607]: cluster 2024-04-25T20:16:57.020902+0000 mgr.x (mgr.14254) 667 : cluster [DBG] pgmap v540: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:16:59.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:16:58 smithi132 bash[18607]: cluster 2024-04-25T20:16:57.020902+0000 mgr.x (mgr.14254) 667 : cluster [DBG] pgmap v540: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:00.644 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:17:01.012 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:00 smithi132 bash[18607]: cluster 2024-04-25T20:16:59.022804+0000 mgr.x (mgr.14254) 668 : cluster [DBG] pgmap v541: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:01.012 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:00 smithi132 bash[18607]: cluster 2024-04-25T20:16:59.022804+0000 mgr.x (mgr.14254) 668 : cluster [DBG] pgmap v541: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:03.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:02 smithi132 bash[18607]: cluster 2024-04-25T20:17:01.023733+0000 mgr.x (mgr.14254) 669 : cluster [DBG] pgmap v542: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:03.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:02 smithi132 bash[18607]: cluster 2024-04-25T20:17:01.023733+0000 mgr.x (mgr.14254) 669 : cluster [DBG] pgmap v542: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:03.721 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:17:03.721 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:17:04.380 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:17:04.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:04 smithi132 bash[18607]: cluster 2024-04-25T20:17:03.025465+0000 mgr.x (mgr.14254) 670 : cluster [DBG] pgmap v543: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:04.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:04 smithi132 bash[18607]: cluster 2024-04-25T20:17:03.025465+0000 mgr.x (mgr.14254) 670 : cluster [DBG] pgmap v543: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:05.381 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:17:05.760 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:05 smithi132 bash[18607]: audit 2024-04-25T20:17:03.725023+0000 mgr.x (mgr.14254) 671 : audit [DBG] from='client.14596 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:17:05.760 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:05 smithi132 bash[18607]: audit 2024-04-25T20:17:03.725023+0000 mgr.x (mgr.14254) 671 : audit [DBG] from='client.14596 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:17:07.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:06 smithi132 bash[18607]: cluster 2024-04-25T20:17:05.027059+0000 mgr.x (mgr.14254) 672 : cluster [DBG] pgmap v544: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:07.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:06 smithi132 bash[18607]: cluster 2024-04-25T20:17:05.027059+0000 mgr.x (mgr.14254) 672 : cluster [DBG] pgmap v544: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:08.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:07 smithi132 bash[18607]: audit 2024-04-25T20:17:06.615152+0000 mon.a (mon.0) 471 : audit [DBG] from='client.? 172.17.0.2:0/562866528' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:17:08.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:07 smithi132 bash[18607]: audit 2024-04-25T20:17:06.615152+0000 mon.a (mon.0) 471 : audit [DBG] from='client.? 172.17.0.2:0/562866528' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:17:09.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:08 smithi132 bash[18607]: cluster 2024-04-25T20:17:07.028578+0000 mgr.x (mgr.14254) 673 : cluster [DBG] pgmap v545: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:09.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:08 smithi132 bash[18607]: cluster 2024-04-25T20:17:07.028578+0000 mgr.x (mgr.14254) 673 : cluster [DBG] pgmap v545: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:10.139 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:17:10.975 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:10 smithi132 bash[18607]: cluster 2024-04-25T20:17:09.030544+0000 mgr.x (mgr.14254) 674 : cluster [DBG] pgmap v546: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:10.975 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:10 smithi132 bash[18607]: cluster 2024-04-25T20:17:09.030544+0000 mgr.x (mgr.14254) 674 : cluster [DBG] pgmap v546: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:12.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:12 smithi132 bash[18607]: cluster 2024-04-25T20:17:11.031347+0000 mgr.x (mgr.14254) 675 : cluster [DBG] pgmap v547: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:12.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:12 smithi132 bash[18607]: cluster 2024-04-25T20:17:11.031347+0000 mgr.x (mgr.14254) 675 : cluster [DBG] pgmap v547: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:12.901 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:17:12.901 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:17:13.609 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:17:14.610 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:17:15.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:14 smithi132 bash[18607]: audit 2024-04-25T20:17:12.904235+0000 mgr.x (mgr.14254) 676 : audit [DBG] from='client.14600 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:17:15.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:14 smithi132 bash[18607]: audit 2024-04-25T20:17:12.904235+0000 mgr.x (mgr.14254) 676 : audit [DBG] from='client.14600 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:17:15.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:14 smithi132 bash[18607]: cluster 2024-04-25T20:17:13.032967+0000 mgr.x (mgr.14254) 677 : cluster [DBG] pgmap v548: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:15.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:14 smithi132 bash[18607]: cluster 2024-04-25T20:17:13.032967+0000 mgr.x (mgr.14254) 677 : cluster [DBG] pgmap v548: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:16.669 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:16 smithi132 bash[18607]: cluster 2024-04-25T20:17:15.034561+0000 mgr.x (mgr.14254) 678 : cluster [DBG] pgmap v549: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:16.669 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:16 smithi132 bash[18607]: cluster 2024-04-25T20:17:15.034561+0000 mgr.x (mgr.14254) 678 : cluster [DBG] pgmap v549: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:19.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:18 smithi132 bash[18607]: cluster 2024-04-25T20:17:17.035956+0000 mgr.x (mgr.14254) 679 : cluster [DBG] pgmap v550: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:19.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:18 smithi132 bash[18607]: cluster 2024-04-25T20:17:17.035956+0000 mgr.x (mgr.14254) 679 : cluster [DBG] pgmap v550: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:19.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:18 smithi132 bash[18607]: audit 2024-04-25T20:17:17.946861+0000 mon.a (mon.0) 472 : audit [DBG] from='client.? 172.17.0.2:0/1443630581' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:17:19.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:18 smithi132 bash[18607]: audit 2024-04-25T20:17:17.946861+0000 mon.a (mon.0) 472 : audit [DBG] from='client.? 172.17.0.2:0/1443630581' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:17:19.370 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:17:21.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:20 smithi132 bash[18607]: cluster 2024-04-25T20:17:19.037984+0000 mgr.x (mgr.14254) 680 : cluster [DBG] pgmap v551: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:21.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:20 smithi132 bash[18607]: cluster 2024-04-25T20:17:19.037984+0000 mgr.x (mgr.14254) 680 : cluster [DBG] pgmap v551: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:22.188 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:17:22.188 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:17:22.890 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:22 smithi132 bash[18607]: cluster 2024-04-25T20:17:21.039452+0000 mgr.x (mgr.14254) 681 : cluster [DBG] pgmap v552: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:22.890 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:22 smithi132 bash[18607]: cluster 2024-04-25T20:17:21.039452+0000 mgr.x (mgr.14254) 681 : cluster [DBG] pgmap v552: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:22.891 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:17:23.892 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:17:23.904 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:23 smithi132 bash[18607]: audit 2024-04-25T20:17:22.189608+0000 mgr.x (mgr.14254) 682 : audit [DBG] from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:17:23.904 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:23 smithi132 bash[18607]: audit 2024-04-25T20:17:22.189608+0000 mgr.x (mgr.14254) 682 : audit [DBG] from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:17:25.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:24 smithi132 bash[18607]: cluster 2024-04-25T20:17:23.041288+0000 mgr.x (mgr.14254) 683 : cluster [DBG] pgmap v553: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:25.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:24 smithi132 bash[18607]: cluster 2024-04-25T20:17:23.041288+0000 mgr.x (mgr.14254) 683 : cluster [DBG] pgmap v553: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:27.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:26 smithi132 bash[18607]: cluster 2024-04-25T20:17:25.042884+0000 mgr.x (mgr.14254) 684 : cluster [DBG] pgmap v554: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:27.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:26 smithi132 bash[18607]: cluster 2024-04-25T20:17:25.042884+0000 mgr.x (mgr.14254) 684 : cluster [DBG] pgmap v554: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:28.649 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:17:29.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:28 smithi132 bash[18607]: cluster 2024-04-25T20:17:27.044292+0000 mgr.x (mgr.14254) 685 : cluster [DBG] pgmap v555: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:29.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:28 smithi132 bash[18607]: cluster 2024-04-25T20:17:27.044292+0000 mgr.x (mgr.14254) 685 : cluster [DBG] pgmap v555: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:30.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:29 smithi132 bash[18607]: audit 2024-04-25T20:17:29.272817+0000 mon.a (mon.0) 473 : audit [DBG] from='client.? 172.17.0.2:0/4157936279' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:17:30.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:29 smithi132 bash[18607]: audit 2024-04-25T20:17:29.272817+0000 mon.a (mon.0) 473 : audit [DBG] from='client.? 172.17.0.2:0/4157936279' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:17:30.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:30 smithi132 bash[18607]: cluster 2024-04-25T20:17:29.046184+0000 mgr.x (mgr.14254) 686 : cluster [DBG] pgmap v556: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:30.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:30 smithi132 bash[18607]: cluster 2024-04-25T20:17:29.046184+0000 mgr.x (mgr.14254) 686 : cluster [DBG] pgmap v556: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:31.532 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:17:31.532 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:17:32.233 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:17:33.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:32 smithi132 bash[18607]: cluster 2024-04-25T20:17:31.047604+0000 mgr.x (mgr.14254) 687 : cluster [DBG] pgmap v557: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:33.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:32 smithi132 bash[18607]: cluster 2024-04-25T20:17:31.047604+0000 mgr.x (mgr.14254) 687 : cluster [DBG] pgmap v557: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:33.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:32 smithi132 bash[18607]: audit 2024-04-25T20:17:31.536416+0000 mgr.x (mgr.14254) 688 : audit [DBG] from='client.14608 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:17:33.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:32 smithi132 bash[18607]: audit 2024-04-25T20:17:31.536416+0000 mgr.x (mgr.14254) 688 : audit [DBG] from='client.14608 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:17:33.233 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:17:35.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:34 smithi132 bash[18607]: cluster 2024-04-25T20:17:33.049414+0000 mgr.x (mgr.14254) 689 : cluster [DBG] pgmap v558: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:35.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:34 smithi132 bash[18607]: cluster 2024-04-25T20:17:33.049414+0000 mgr.x (mgr.14254) 689 : cluster [DBG] pgmap v558: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:37.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:36 smithi132 bash[18607]: cluster 2024-04-25T20:17:35.051288+0000 mgr.x (mgr.14254) 690 : cluster [DBG] pgmap v559: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:37.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:36 smithi132 bash[18607]: cluster 2024-04-25T20:17:35.051288+0000 mgr.x (mgr.14254) 690 : cluster [DBG] pgmap v559: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:37.997 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:17:38.871 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:38 smithi132 bash[18607]: cluster 2024-04-25T20:17:37.052811+0000 mgr.x (mgr.14254) 691 : cluster [DBG] pgmap v560: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:38.871 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:38 smithi132 bash[18607]: cluster 2024-04-25T20:17:37.052811+0000 mgr.x (mgr.14254) 691 : cluster [DBG] pgmap v560: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:40.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:40 smithi132 bash[18607]: cluster 2024-04-25T20:17:39.053757+0000 mgr.x (mgr.14254) 692 : cluster [DBG] pgmap v561: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:40.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:40 smithi132 bash[18607]: cluster 2024-04-25T20:17:39.053757+0000 mgr.x (mgr.14254) 692 : cluster [DBG] pgmap v561: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:40.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:40 smithi132 bash[18607]: audit 2024-04-25T20:17:40.608903+0000 mon.a (mon.0) 474 : audit [DBG] from='client.? 172.17.0.2:0/4161847292' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:17:40.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:40 smithi132 bash[18607]: audit 2024-04-25T20:17:40.608903+0000 mon.a (mon.0) 474 : audit [DBG] from='client.? 172.17.0.2:0/4161847292' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:17:40.855 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:17:40.855 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:17:41.592 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:17:42.594 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:17:43.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:42 smithi132 bash[18607]: audit 2024-04-25T20:17:40.856102+0000 mgr.x (mgr.14254) 693 : audit [DBG] from='client.14612 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:17:43.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:42 smithi132 bash[18607]: audit 2024-04-25T20:17:40.856102+0000 mgr.x (mgr.14254) 693 : audit [DBG] from='client.14612 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:17:43.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:42 smithi132 bash[18607]: cluster 2024-04-25T20:17:41.055249+0000 mgr.x (mgr.14254) 694 : cluster [DBG] pgmap v562: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:43.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:42 smithi132 bash[18607]: cluster 2024-04-25T20:17:41.055249+0000 mgr.x (mgr.14254) 694 : cluster [DBG] pgmap v562: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:45.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:44 smithi132 bash[18607]: cluster 2024-04-25T20:17:43.056959+0000 mgr.x (mgr.14254) 695 : cluster [DBG] pgmap v563: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:45.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:44 smithi132 bash[18607]: cluster 2024-04-25T20:17:43.056959+0000 mgr.x (mgr.14254) 695 : cluster [DBG] pgmap v563: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:47.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:46 smithi132 bash[18607]: cluster 2024-04-25T20:17:45.058515+0000 mgr.x (mgr.14254) 696 : cluster [DBG] pgmap v564: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:47.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:46 smithi132 bash[18607]: cluster 2024-04-25T20:17:45.058515+0000 mgr.x (mgr.14254) 696 : cluster [DBG] pgmap v564: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:47.358 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:17:49.010 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:48 smithi132 bash[18607]: cluster 2024-04-25T20:17:47.059913+0000 mgr.x (mgr.14254) 697 : cluster [DBG] pgmap v565: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:49.011 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:48 smithi132 bash[18607]: cluster 2024-04-25T20:17:47.059913+0000 mgr.x (mgr.14254) 697 : cluster [DBG] pgmap v565: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:50.193 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:17:50.194 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:17:50.833 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:17:50.909 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:50 smithi132 bash[18607]: cluster 2024-04-25T20:17:49.061522+0000 mgr.x (mgr.14254) 698 : cluster [DBG] pgmap v566: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:50.909 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:50 smithi132 bash[18607]: cluster 2024-04-25T20:17:49.061522+0000 mgr.x (mgr.14254) 698 : cluster [DBG] pgmap v566: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:51.834 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:17:51.971 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:51 smithi132 bash[18607]: audit 2024-04-25T20:17:50.194818+0000 mgr.x (mgr.14254) 699 : audit [DBG] from='client.14614 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:17:51.971 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:51 smithi132 bash[18607]: audit 2024-04-25T20:17:50.194818+0000 mgr.x (mgr.14254) 699 : audit [DBG] from='client.14614 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:17:53.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:52 smithi132 bash[18607]: cluster 2024-04-25T20:17:51.062933+0000 mgr.x (mgr.14254) 700 : cluster [DBG] pgmap v567: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:53.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:52 smithi132 bash[18607]: cluster 2024-04-25T20:17:51.062933+0000 mgr.x (mgr.14254) 700 : cluster [DBG] pgmap v567: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:53.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:52 smithi132 bash[18607]: audit 2024-04-25T20:17:51.937541+0000 mon.a (mon.0) 475 : audit [DBG] from='client.? 172.17.0.2:0/108611148' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:17:53.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:52 smithi132 bash[18607]: audit 2024-04-25T20:17:51.937541+0000 mon.a (mon.0) 475 : audit [DBG] from='client.? 172.17.0.2:0/108611148' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:17:54.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:53 smithi132 bash[18607]: cluster 2024-04-25T20:17:53.064070+0000 mgr.x (mgr.14254) 701 : cluster [DBG] pgmap v568: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:54.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:53 smithi132 bash[18607]: cluster 2024-04-25T20:17:53.064070+0000 mgr.x (mgr.14254) 701 : cluster [DBG] pgmap v568: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:56.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:56 smithi132 bash[18607]: cluster 2024-04-25T20:17:55.065665+0000 mgr.x (mgr.14254) 702 : cluster [DBG] pgmap v569: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:56.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:56 smithi132 bash[18607]: cluster 2024-04-25T20:17:55.065665+0000 mgr.x (mgr.14254) 702 : cluster [DBG] pgmap v569: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:56.594 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:17:58.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:58 smithi132 bash[18607]: cluster 2024-04-25T20:17:57.067008+0000 mgr.x (mgr.14254) 703 : cluster [DBG] pgmap v570: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:58.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:17:58 smithi132 bash[18607]: cluster 2024-04-25T20:17:57.067008+0000 mgr.x (mgr.14254) 703 : cluster [DBG] pgmap v570: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:17:59.377 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:17:59.377 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:18:00.042 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:18:00.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:00 smithi132 bash[18607]: cluster 2024-04-25T20:17:59.068222+0000 mgr.x (mgr.14254) 704 : cluster [DBG] pgmap v571: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:00.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:00 smithi132 bash[18607]: cluster 2024-04-25T20:17:59.068222+0000 mgr.x (mgr.14254) 704 : cluster [DBG] pgmap v571: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:00.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:00 smithi132 bash[18607]: audit 2024-04-25T20:17:59.378398+0000 mgr.x (mgr.14254) 705 : audit [DBG] from='client.14618 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:18:00.519 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:00 smithi132 bash[18607]: audit 2024-04-25T20:17:59.378398+0000 mgr.x (mgr.14254) 705 : audit [DBG] from='client.14618 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:18:01.043 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:18:02.234 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:02 smithi132 bash[18607]: cluster 2024-04-25T20:18:01.069112+0000 mgr.x (mgr.14254) 706 : cluster [DBG] pgmap v572: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:02.234 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:02 smithi132 bash[18607]: cluster 2024-04-25T20:18:01.069112+0000 mgr.x (mgr.14254) 706 : cluster [DBG] pgmap v572: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:04.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:04 smithi132 bash[18607]: cluster 2024-04-25T20:18:03.070852+0000 mgr.x (mgr.14254) 707 : cluster [DBG] pgmap v573: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:04.519 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:04 smithi132 bash[18607]: cluster 2024-04-25T20:18:03.070852+0000 mgr.x (mgr.14254) 707 : cluster [DBG] pgmap v573: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:04.519 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:04 smithi132 bash[18607]: audit 2024-04-25T20:18:03.264144+0000 mon.a (mon.0) 476 : audit [DBG] from='client.? 172.17.0.2:0/2745524944' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:18:04.519 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:04 smithi132 bash[18607]: audit 2024-04-25T20:18:03.264144+0000 mon.a (mon.0) 476 : audit [DBG] from='client.? 172.17.0.2:0/2745524944' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:18:05.795 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:18:06.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:06 smithi132 bash[18607]: cluster 2024-04-25T20:18:05.072490+0000 mgr.x (mgr.14254) 708 : cluster [DBG] pgmap v574: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:06.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:06 smithi132 bash[18607]: cluster 2024-04-25T20:18:05.072490+0000 mgr.x (mgr.14254) 708 : cluster [DBG] pgmap v574: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:08.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:08 smithi132 bash[18607]: cluster 2024-04-25T20:18:07.073916+0000 mgr.x (mgr.14254) 709 : cluster [DBG] pgmap v575: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:08.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:08 smithi132 bash[18607]: cluster 2024-04-25T20:18:07.073916+0000 mgr.x (mgr.14254) 709 : cluster [DBG] pgmap v575: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:08.653 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:18:08.654 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:18:09.319 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:18:10.320 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:18:10.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:10 smithi132 bash[18607]: audit 2024-04-25T20:18:08.656561+0000 mgr.x (mgr.14254) 710 : audit [DBG] from='client.14622 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:18:10.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:10 smithi132 bash[18607]: audit 2024-04-25T20:18:08.656561+0000 mgr.x (mgr.14254) 710 : audit [DBG] from='client.14622 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:18:10.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:10 smithi132 bash[18607]: cluster 2024-04-25T20:18:09.075758+0000 mgr.x (mgr.14254) 711 : cluster [DBG] pgmap v576: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:10.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:10 smithi132 bash[18607]: cluster 2024-04-25T20:18:09.075758+0000 mgr.x (mgr.14254) 711 : cluster [DBG] pgmap v576: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:12.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:12 smithi132 bash[18607]: cluster 2024-04-25T20:18:11.077179+0000 mgr.x (mgr.14254) 712 : cluster [DBG] pgmap v577: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:12.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:12 smithi132 bash[18607]: cluster 2024-04-25T20:18:11.077179+0000 mgr.x (mgr.14254) 712 : cluster [DBG] pgmap v577: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:14.519 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:14 smithi132 bash[18607]: cluster 2024-04-25T20:18:13.078009+0000 mgr.x (mgr.14254) 713 : cluster [DBG] pgmap v578: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:14.519 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:14 smithi132 bash[18607]: cluster 2024-04-25T20:18:13.078009+0000 mgr.x (mgr.14254) 713 : cluster [DBG] pgmap v578: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:15.079 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:18:16.268 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:16 smithi132 bash[18607]: cluster 2024-04-25T20:18:15.079385+0000 mgr.x (mgr.14254) 714 : cluster [DBG] pgmap v579: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:16.268 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:16 smithi132 bash[18607]: cluster 2024-04-25T20:18:15.079385+0000 mgr.x (mgr.14254) 714 : cluster [DBG] pgmap v579: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:17.870 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:18:17.870 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:18:18.268 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:18 smithi132 bash[18607]: cluster 2024-04-25T20:18:17.080845+0000 mgr.x (mgr.14254) 715 : cluster [DBG] pgmap v580: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:18.268 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:18 smithi132 bash[18607]: cluster 2024-04-25T20:18:17.080845+0000 mgr.x (mgr.14254) 715 : cluster [DBG] pgmap v580: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:18.497 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:18:19.498 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:18:19.509 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:19 smithi132 bash[18607]: audit 2024-04-25T20:18:17.871156+0000 mgr.x (mgr.14254) 716 : audit [DBG] from='client.14624 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:18:19.509 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:19 smithi132 bash[18607]: audit 2024-04-25T20:18:17.871156+0000 mgr.x (mgr.14254) 716 : audit [DBG] from='client.14624 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:18:19.510 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:19 smithi132 bash[18607]: audit 2024-04-25T20:18:18.603899+0000 mon.a (mon.0) 477 : audit [DBG] from='client.? 172.17.0.2:0/2885783046' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:18:19.510 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:19 smithi132 bash[18607]: audit 2024-04-25T20:18:18.603899+0000 mon.a (mon.0) 477 : audit [DBG] from='client.? 172.17.0.2:0/2885783046' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:18:20.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:20 smithi132 bash[18607]: cluster 2024-04-25T20:18:19.082647+0000 mgr.x (mgr.14254) 717 : cluster [DBG] pgmap v581: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:20.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:20 smithi132 bash[18607]: cluster 2024-04-25T20:18:19.082647+0000 mgr.x (mgr.14254) 717 : cluster [DBG] pgmap v581: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:22.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:22 smithi132 bash[18607]: cluster 2024-04-25T20:18:21.084073+0000 mgr.x (mgr.14254) 718 : cluster [DBG] pgmap v582: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:22.519 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:22 smithi132 bash[18607]: cluster 2024-04-25T20:18:21.084073+0000 mgr.x (mgr.14254) 718 : cluster [DBG] pgmap v582: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:24.260 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:18:24.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:24 smithi132 bash[18607]: cluster 2024-04-25T20:18:23.085908+0000 mgr.x (mgr.14254) 719 : cluster [DBG] pgmap v583: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:24.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:24 smithi132 bash[18607]: cluster 2024-04-25T20:18:23.085908+0000 mgr.x (mgr.14254) 719 : cluster [DBG] pgmap v583: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:26.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:26 smithi132 bash[18607]: cluster 2024-04-25T20:18:25.087397+0000 mgr.x (mgr.14254) 720 : cluster [DBG] pgmap v584: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:26.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:26 smithi132 bash[18607]: cluster 2024-04-25T20:18:25.087397+0000 mgr.x (mgr.14254) 720 : cluster [DBG] pgmap v584: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:26.926 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:18:26.927 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:18:27.627 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:18:28.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:28 smithi132 bash[18607]: audit 2024-04-25T20:18:26.927365+0000 mgr.x (mgr.14254) 721 : audit [DBG] from='client.14628 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:18:28.519 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:28 smithi132 bash[18607]: audit 2024-04-25T20:18:26.927365+0000 mgr.x (mgr.14254) 721 : audit [DBG] from='client.14628 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:18:28.519 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:28 smithi132 bash[18607]: cluster 2024-04-25T20:18:27.088821+0000 mgr.x (mgr.14254) 722 : cluster [DBG] pgmap v585: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:28.519 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:28 smithi132 bash[18607]: cluster 2024-04-25T20:18:27.088821+0000 mgr.x (mgr.14254) 722 : cluster [DBG] pgmap v585: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:28.628 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:18:30.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:30 smithi132 bash[18607]: cluster 2024-04-25T20:18:29.090694+0000 mgr.x (mgr.14254) 723 : cluster [DBG] pgmap v586: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:30.519 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:30 smithi132 bash[18607]: cluster 2024-04-25T20:18:29.090694+0000 mgr.x (mgr.14254) 723 : cluster [DBG] pgmap v586: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:32.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:32 smithi132 bash[18607]: cluster 2024-04-25T20:18:31.092100+0000 mgr.x (mgr.14254) 724 : cluster [DBG] pgmap v587: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:32.519 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:32 smithi132 bash[18607]: cluster 2024-04-25T20:18:31.092100+0000 mgr.x (mgr.14254) 724 : cluster [DBG] pgmap v587: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:33.386 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:18:34.268 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:34 smithi132 bash[18607]: cluster 2024-04-25T20:18:33.093931+0000 mgr.x (mgr.14254) 725 : cluster [DBG] pgmap v588: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:34.269 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:34 smithi132 bash[18607]: cluster 2024-04-25T20:18:33.093931+0000 mgr.x (mgr.14254) 725 : cluster [DBG] pgmap v588: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:34.269 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:34 smithi132 bash[18607]: audit 2024-04-25T20:18:33.932050+0000 mon.a (mon.0) 478 : audit [DBG] from='client.? 172.17.0.2:0/3856282292' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:18:34.269 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:34 smithi132 bash[18607]: audit 2024-04-25T20:18:33.932050+0000 mon.a (mon.0) 478 : audit [DBG] from='client.? 172.17.0.2:0/3856282292' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:18:36.285 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:18:36.285 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:18:36.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:36 smithi132 bash[18607]: cluster 2024-04-25T20:18:35.094664+0000 mgr.x (mgr.14254) 726 : cluster [DBG] pgmap v589: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:36.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:36 smithi132 bash[18607]: cluster 2024-04-25T20:18:35.094664+0000 mgr.x (mgr.14254) 726 : cluster [DBG] pgmap v589: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:37.088 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:18:37.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:37 smithi132 bash[18607]: audit 2024-04-25T20:18:36.285704+0000 mgr.x (mgr.14254) 727 : audit [DBG] from='client.14632 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:18:37.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:37 smithi132 bash[18607]: audit 2024-04-25T20:18:36.285704+0000 mgr.x (mgr.14254) 727 : audit [DBG] from='client.14632 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:18:38.089 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:18:38.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:38 smithi132 bash[18607]: cluster 2024-04-25T20:18:37.096159+0000 mgr.x (mgr.14254) 728 : cluster [DBG] pgmap v590: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:38.519 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:38 smithi132 bash[18607]: cluster 2024-04-25T20:18:37.096159+0000 mgr.x (mgr.14254) 728 : cluster [DBG] pgmap v590: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:40.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:40 smithi132 bash[18607]: cluster 2024-04-25T20:18:39.098072+0000 mgr.x (mgr.14254) 729 : cluster [DBG] pgmap v591: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:40.519 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:40 smithi132 bash[18607]: cluster 2024-04-25T20:18:39.098072+0000 mgr.x (mgr.14254) 729 : cluster [DBG] pgmap v591: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:42.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:42 smithi132 bash[18607]: cluster 2024-04-25T20:18:41.099477+0000 mgr.x (mgr.14254) 730 : cluster [DBG] pgmap v592: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:42.519 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:42 smithi132 bash[18607]: cluster 2024-04-25T20:18:41.099477+0000 mgr.x (mgr.14254) 730 : cluster [DBG] pgmap v592: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:42.849 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:18:44.233 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:44 smithi132 bash[18607]: cluster 2024-04-25T20:18:43.101263+0000 mgr.x (mgr.14254) 731 : cluster [DBG] pgmap v593: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:44.234 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:44 smithi132 bash[18607]: cluster 2024-04-25T20:18:43.101263+0000 mgr.x (mgr.14254) 731 : cluster [DBG] pgmap v593: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:45.781 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:18:45.781 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:18:46.416 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:18:46.519 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:46 smithi132 bash[18607]: cluster 2024-04-25T20:18:45.102657+0000 mgr.x (mgr.14254) 732 : cluster [DBG] pgmap v594: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:46.519 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:46 smithi132 bash[18607]: cluster 2024-04-25T20:18:45.102657+0000 mgr.x (mgr.14254) 732 : cluster [DBG] pgmap v594: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:47.418 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:18:47.429 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:47 smithi132 bash[18607]: audit 2024-04-25T20:18:45.781355+0000 mgr.x (mgr.14254) 733 : audit [DBG] from='client.14634 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:18:47.429 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:47 smithi132 bash[18607]: audit 2024-04-25T20:18:45.781355+0000 mgr.x (mgr.14254) 733 : audit [DBG] from='client.14634 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:18:48.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:48 smithi132 bash[18607]: cluster 2024-04-25T20:18:47.104052+0000 mgr.x (mgr.14254) 734 : cluster [DBG] pgmap v595: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:48.519 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:48 smithi132 bash[18607]: cluster 2024-04-25T20:18:47.104052+0000 mgr.x (mgr.14254) 734 : cluster [DBG] pgmap v595: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:50.519 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:50 smithi132 bash[18607]: cluster 2024-04-25T20:18:49.105921+0000 mgr.x (mgr.14254) 735 : cluster [DBG] pgmap v596: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:50.519 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:50 smithi132 bash[18607]: cluster 2024-04-25T20:18:49.105921+0000 mgr.x (mgr.14254) 735 : cluster [DBG] pgmap v596: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:50.519 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:50 smithi132 bash[18607]: audit 2024-04-25T20:18:49.265615+0000 mon.a (mon.0) 479 : audit [DBG] from='client.? 172.17.0.2:0/614820096' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:18:50.519 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:50 smithi132 bash[18607]: audit 2024-04-25T20:18:49.265615+0000 mon.a (mon.0) 479 : audit [DBG] from='client.? 172.17.0.2:0/614820096' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:18:52.175 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:18:52.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:52 smithi132 bash[18607]: cluster 2024-04-25T20:18:51.107333+0000 mgr.x (mgr.14254) 736 : cluster [DBG] pgmap v597: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:52.519 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:52 smithi132 bash[18607]: cluster 2024-04-25T20:18:51.107333+0000 mgr.x (mgr.14254) 736 : cluster [DBG] pgmap v597: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:54.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:54 smithi132 bash[18607]: cluster 2024-04-25T20:18:53.108304+0000 mgr.x (mgr.14254) 737 : cluster [DBG] pgmap v598: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:54.519 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:54 smithi132 bash[18607]: cluster 2024-04-25T20:18:53.108304+0000 mgr.x (mgr.14254) 737 : cluster [DBG] pgmap v598: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:54.925 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:18:54.926 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:18:55.651 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:18:56.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:56 smithi132 bash[18607]: audit 2024-04-25T20:18:54.926432+0000 mgr.x (mgr.14254) 738 : audit [DBG] from='client.14638 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:18:56.519 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:56 smithi132 bash[18607]: audit 2024-04-25T20:18:54.926432+0000 mgr.x (mgr.14254) 738 : audit [DBG] from='client.14638 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:18:56.519 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:56 smithi132 bash[18607]: cluster 2024-04-25T20:18:55.109876+0000 mgr.x (mgr.14254) 739 : cluster [DBG] pgmap v599: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:56.519 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:56 smithi132 bash[18607]: cluster 2024-04-25T20:18:55.109876+0000 mgr.x (mgr.14254) 739 : cluster [DBG] pgmap v599: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:56.652 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:18:58.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:58 smithi132 bash[18607]: cluster 2024-04-25T20:18:57.111291+0000 mgr.x (mgr.14254) 740 : cluster [DBG] pgmap v600: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:18:58.519 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:18:58 smithi132 bash[18607]: cluster 2024-04-25T20:18:57.111291+0000 mgr.x (mgr.14254) 740 : cluster [DBG] pgmap v600: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:00.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:00 smithi132 bash[18607]: cluster 2024-04-25T20:18:59.113171+0000 mgr.x (mgr.14254) 741 : cluster [DBG] pgmap v601: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:00.519 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:00 smithi132 bash[18607]: cluster 2024-04-25T20:18:59.113171+0000 mgr.x (mgr.14254) 741 : cluster [DBG] pgmap v601: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:01.404 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:19:02.452 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:02 smithi132 bash[18607]: cluster 2024-04-25T20:19:01.114575+0000 mgr.x (mgr.14254) 742 : cluster [DBG] pgmap v602: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:02.452 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:02 smithi132 bash[18607]: cluster 2024-04-25T20:19:01.114575+0000 mgr.x (mgr.14254) 742 : cluster [DBG] pgmap v602: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:04.205 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:19:04.205 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:19:04.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:04 smithi132 bash[18607]: cluster 2024-04-25T20:19:03.115559+0000 mgr.x (mgr.14254) 743 : cluster [DBG] pgmap v603: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:04.519 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:04 smithi132 bash[18607]: cluster 2024-04-25T20:19:03.115559+0000 mgr.x (mgr.14254) 743 : cluster [DBG] pgmap v603: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:04.904 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:19:05.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:05 smithi132 bash[18607]: audit 2024-04-25T20:19:04.209333+0000 mgr.x (mgr.14254) 744 : audit [DBG] from='client.14640 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:19:05.519 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:05 smithi132 bash[18607]: audit 2024-04-25T20:19:04.209333+0000 mgr.x (mgr.14254) 744 : audit [DBG] from='client.14640 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:19:05.519 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:05 smithi132 bash[18607]: audit 2024-04-25T20:19:04.602891+0000 mon.a (mon.0) 480 : audit [DBG] from='client.? 172.17.0.2:0/1117550540' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:19:05.519 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:05 smithi132 bash[18607]: audit 2024-04-25T20:19:04.602891+0000 mon.a (mon.0) 480 : audit [DBG] from='client.? 172.17.0.2:0/1117550540' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:19:05.905 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:19:06.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:06 smithi132 bash[18607]: cluster 2024-04-25T20:19:05.117008+0000 mgr.x (mgr.14254) 745 : cluster [DBG] pgmap v604: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:06.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:06 smithi132 bash[18607]: cluster 2024-04-25T20:19:05.117008+0000 mgr.x (mgr.14254) 745 : cluster [DBG] pgmap v604: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:08.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:08 smithi132 bash[18607]: cluster 2024-04-25T20:19:07.118077+0000 mgr.x (mgr.14254) 746 : cluster [DBG] pgmap v605: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:08.519 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:08 smithi132 bash[18607]: cluster 2024-04-25T20:19:07.118077+0000 mgr.x (mgr.14254) 746 : cluster [DBG] pgmap v605: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:10.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:10 smithi132 bash[18607]: cluster 2024-04-25T20:19:09.119968+0000 mgr.x (mgr.14254) 747 : cluster [DBG] pgmap v606: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:10.519 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:10 smithi132 bash[18607]: cluster 2024-04-25T20:19:09.119968+0000 mgr.x (mgr.14254) 747 : cluster [DBG] pgmap v606: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:10.659 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:19:12.518 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:12 smithi132 bash[18607]: cluster 2024-04-25T20:19:11.121454+0000 mgr.x (mgr.14254) 748 : cluster [DBG] pgmap v607: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:12.519 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:12 smithi132 bash[18607]: cluster 2024-04-25T20:19:11.121454+0000 mgr.x (mgr.14254) 748 : cluster [DBG] pgmap v607: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:13.378 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:19:13.378 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:19:14.062 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:19:14.906 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:14 smithi132 bash[18607]: cluster 2024-04-25T20:19:13.123245+0000 mgr.x (mgr.14254) 749 : cluster [DBG] pgmap v608: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:14.906 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:14 smithi132 bash[18607]: cluster 2024-04-25T20:19:13.123245+0000 mgr.x (mgr.14254) 749 : cluster [DBG] pgmap v608: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:14.906 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:14 smithi132 bash[18607]: audit 2024-04-25T20:19:13.382024+0000 mgr.x (mgr.14254) 750 : audit [DBG] from='client.14644 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:19:14.906 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:14 smithi132 bash[18607]: audit 2024-04-25T20:19:13.382024+0000 mgr.x (mgr.14254) 750 : audit [DBG] from='client.14644 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:19:15.062 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:19:17.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:16 smithi132 bash[18607]: cluster 2024-04-25T20:19:15.124183+0000 mgr.x (mgr.14254) 751 : cluster [DBG] pgmap v609: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:17.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:16 smithi132 bash[18607]: cluster 2024-04-25T20:19:15.124183+0000 mgr.x (mgr.14254) 751 : cluster [DBG] pgmap v609: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:19.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:18 smithi132 bash[18607]: cluster 2024-04-25T20:19:17.125632+0000 mgr.x (mgr.14254) 752 : cluster [DBG] pgmap v610: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:19.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:18 smithi132 bash[18607]: cluster 2024-04-25T20:19:17.125632+0000 mgr.x (mgr.14254) 752 : cluster [DBG] pgmap v610: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:19.826 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:19:20.821 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:20 smithi132 bash[18607]: cluster 2024-04-25T20:19:19.127568+0000 mgr.x (mgr.14254) 753 : cluster [DBG] pgmap v611: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:20.822 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:20 smithi132 bash[18607]: cluster 2024-04-25T20:19:19.127568+0000 mgr.x (mgr.14254) 753 : cluster [DBG] pgmap v611: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:20.822 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:20 smithi132 bash[18607]: audit 2024-04-25T20:19:19.933150+0000 mon.a (mon.0) 481 : audit [DBG] from='client.? 172.17.0.2:0/3650633726' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:19:20.822 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:20 smithi132 bash[18607]: audit 2024-04-25T20:19:19.933150+0000 mon.a (mon.0) 481 : audit [DBG] from='client.? 172.17.0.2:0/3650633726' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:19:22.765 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:19:22.765 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:19:22.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:22 smithi132 bash[18607]: cluster 2024-04-25T20:19:21.129107+0000 mgr.x (mgr.14254) 754 : cluster [DBG] pgmap v612: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:22.768 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:22 smithi132 bash[18607]: cluster 2024-04-25T20:19:21.129107+0000 mgr.x (mgr.14254) 754 : cluster [DBG] pgmap v612: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:23.380 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:19:24.382 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:19:24.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:24 smithi132 bash[18607]: audit 2024-04-25T20:19:22.765474+0000 mgr.x (mgr.14254) 755 : audit [DBG] from='client.14648 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:19:24.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:24 smithi132 bash[18607]: audit 2024-04-25T20:19:22.765474+0000 mgr.x (mgr.14254) 755 : audit [DBG] from='client.14648 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:19:24.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:24 smithi132 bash[18607]: cluster 2024-04-25T20:19:23.130878+0000 mgr.x (mgr.14254) 756 : cluster [DBG] pgmap v613: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:24.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:24 smithi132 bash[18607]: cluster 2024-04-25T20:19:23.130878+0000 mgr.x (mgr.14254) 756 : cluster [DBG] pgmap v613: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:27.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:26 smithi132 bash[18607]: cluster 2024-04-25T20:19:25.132295+0000 mgr.x (mgr.14254) 757 : cluster [DBG] pgmap v614: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:27.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:26 smithi132 bash[18607]: cluster 2024-04-25T20:19:25.132295+0000 mgr.x (mgr.14254) 757 : cluster [DBG] pgmap v614: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:29.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:28 smithi132 bash[18607]: cluster 2024-04-25T20:19:27.133048+0000 mgr.x (mgr.14254) 758 : cluster [DBG] pgmap v615: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:29.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:28 smithi132 bash[18607]: cluster 2024-04-25T20:19:27.133048+0000 mgr.x (mgr.14254) 758 : cluster [DBG] pgmap v615: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:29.142 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:19:30.872 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:30 smithi132 bash[18607]: cluster 2024-04-25T20:19:29.135221+0000 mgr.x (mgr.14254) 759 : cluster [DBG] pgmap v616: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:30.873 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:30 smithi132 bash[18607]: cluster 2024-04-25T20:19:29.135221+0000 mgr.x (mgr.14254) 759 : cluster [DBG] pgmap v616: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:32.053 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:19:32.054 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:19:32.714 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:19:33.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:32 smithi132 bash[18607]: cluster 2024-04-25T20:19:31.136781+0000 mgr.x (mgr.14254) 760 : cluster [DBG] pgmap v617: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:33.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:32 smithi132 bash[18607]: cluster 2024-04-25T20:19:31.136781+0000 mgr.x (mgr.14254) 760 : cluster [DBG] pgmap v617: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:33.716 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:19:34.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:33 smithi132 bash[18607]: audit 2024-04-25T20:19:32.056420+0000 mgr.x (mgr.14254) 761 : audit [DBG] from='client.14650 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:19:34.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:33 smithi132 bash[18607]: audit 2024-04-25T20:19:32.056420+0000 mgr.x (mgr.14254) 761 : audit [DBG] from='client.14650 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:19:35.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:34 smithi132 bash[18607]: cluster 2024-04-25T20:19:33.138700+0000 mgr.x (mgr.14254) 762 : cluster [DBG] pgmap v618: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:35.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:34 smithi132 bash[18607]: cluster 2024-04-25T20:19:33.138700+0000 mgr.x (mgr.14254) 762 : cluster [DBG] pgmap v618: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:36.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:35 smithi132 bash[18607]: audit 2024-04-25T20:19:35.266533+0000 mon.a (mon.0) 482 : audit [DBG] from='client.? 172.17.0.2:0/3371478836' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:19:36.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:35 smithi132 bash[18607]: audit 2024-04-25T20:19:35.266533+0000 mon.a (mon.0) 482 : audit [DBG] from='client.? 172.17.0.2:0/3371478836' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:19:37.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:36 smithi132 bash[18607]: cluster 2024-04-25T20:19:35.140300+0000 mgr.x (mgr.14254) 763 : cluster [DBG] pgmap v619: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:37.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:36 smithi132 bash[18607]: cluster 2024-04-25T20:19:35.140300+0000 mgr.x (mgr.14254) 763 : cluster [DBG] pgmap v619: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:38.477 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:19:38.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:38 smithi132 bash[18607]: cluster 2024-04-25T20:19:37.141715+0000 mgr.x (mgr.14254) 764 : cluster [DBG] pgmap v620: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:38.769 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:38 smithi132 bash[18607]: cluster 2024-04-25T20:19:37.141715+0000 mgr.x (mgr.14254) 764 : cluster [DBG] pgmap v620: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:40.886 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:40 smithi132 bash[18607]: cluster 2024-04-25T20:19:39.143578+0000 mgr.x (mgr.14254) 765 : cluster [DBG] pgmap v621: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:40.886 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:40 smithi132 bash[18607]: cluster 2024-04-25T20:19:39.143578+0000 mgr.x (mgr.14254) 765 : cluster [DBG] pgmap v621: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:41.276 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:19:41.277 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:19:41.959 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:19:42.960 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:19:42.971 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:42 smithi132 bash[18607]: cluster 2024-04-25T20:19:41.144412+0000 mgr.x (mgr.14254) 766 : cluster [DBG] pgmap v622: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:42.971 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:42 smithi132 bash[18607]: cluster 2024-04-25T20:19:41.144412+0000 mgr.x (mgr.14254) 766 : cluster [DBG] pgmap v622: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:42.971 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:42 smithi132 bash[18607]: audit 2024-04-25T20:19:41.279440+0000 mgr.x (mgr.14254) 767 : audit [DBG] from='client.14654 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:19:42.971 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:42 smithi132 bash[18607]: audit 2024-04-25T20:19:41.279440+0000 mgr.x (mgr.14254) 767 : audit [DBG] from='client.14654 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:19:45.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:44 smithi132 bash[18607]: cluster 2024-04-25T20:19:43.145610+0000 mgr.x (mgr.14254) 768 : cluster [DBG] pgmap v623: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:45.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:44 smithi132 bash[18607]: cluster 2024-04-25T20:19:43.145610+0000 mgr.x (mgr.14254) 768 : cluster [DBG] pgmap v623: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:47.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:46 smithi132 bash[18607]: cluster 2024-04-25T20:19:45.146982+0000 mgr.x (mgr.14254) 769 : cluster [DBG] pgmap v624: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:47.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:46 smithi132 bash[18607]: cluster 2024-04-25T20:19:45.146982+0000 mgr.x (mgr.14254) 769 : cluster [DBG] pgmap v624: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:47.719 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:19:48.960 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:48 smithi132 bash[18607]: cluster 2024-04-25T20:19:47.148355+0000 mgr.x (mgr.14254) 770 : cluster [DBG] pgmap v625: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:48.960 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:48 smithi132 bash[18607]: cluster 2024-04-25T20:19:47.148355+0000 mgr.x (mgr.14254) 770 : cluster [DBG] pgmap v625: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:50.542 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:19:50.542 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:19:51.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:50 smithi132 bash[18607]: cluster 2024-04-25T20:19:49.150256+0000 mgr.x (mgr.14254) 771 : cluster [DBG] pgmap v626: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:51.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:50 smithi132 bash[18607]: cluster 2024-04-25T20:19:49.150256+0000 mgr.x (mgr.14254) 771 : cluster [DBG] pgmap v626: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:51.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:50 smithi132 bash[18607]: audit 2024-04-25T20:19:50.589548+0000 mon.a (mon.0) 483 : audit [DBG] from='client.? 172.17.0.2:0/1866086644' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:19:51.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:50 smithi132 bash[18607]: audit 2024-04-25T20:19:50.589548+0000 mon.a (mon.0) 483 : audit [DBG] from='client.? 172.17.0.2:0/1866086644' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:19:51.280 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:19:52.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:51 smithi132 bash[18607]: audit 2024-04-25T20:19:50.544827+0000 mgr.x (mgr.14254) 772 : audit [DBG] from='client.14656 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:19:52.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:51 smithi132 bash[18607]: audit 2024-04-25T20:19:50.544827+0000 mgr.x (mgr.14254) 772 : audit [DBG] from='client.14656 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:19:52.282 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:19:53.018 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:52 smithi132 bash[18607]: cluster 2024-04-25T20:19:51.151663+0000 mgr.x (mgr.14254) 773 : cluster [DBG] pgmap v627: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:53.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:52 smithi132 bash[18607]: cluster 2024-04-25T20:19:51.151663+0000 mgr.x (mgr.14254) 773 : cluster [DBG] pgmap v627: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:55.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:54 smithi132 bash[18607]: cluster 2024-04-25T20:19:53.153494+0000 mgr.x (mgr.14254) 774 : cluster [DBG] pgmap v628: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:55.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:54 smithi132 bash[18607]: cluster 2024-04-25T20:19:53.153494+0000 mgr.x (mgr.14254) 774 : cluster [DBG] pgmap v628: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:57.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:56 smithi132 bash[18607]: cluster 2024-04-25T20:19:55.154909+0000 mgr.x (mgr.14254) 775 : cluster [DBG] pgmap v629: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:57.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:56 smithi132 bash[18607]: cluster 2024-04-25T20:19:55.154909+0000 mgr.x (mgr.14254) 775 : cluster [DBG] pgmap v629: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:57.031 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:19:59.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:58 smithi132 bash[18607]: cluster 2024-04-25T20:19:57.156298+0000 mgr.x (mgr.14254) 776 : cluster [DBG] pgmap v630: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:59.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:19:58 smithi132 bash[18607]: cluster 2024-04-25T20:19:57.156298+0000 mgr.x (mgr.14254) 776 : cluster [DBG] pgmap v630: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:19:59.868 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:19:59.869 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:20:00.573 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:20:00.889 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:00 smithi132 bash[18607]: cluster 2024-04-25T20:19:59.158235+0000 mgr.x (mgr.14254) 777 : cluster [DBG] pgmap v631: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:00.889 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:00 smithi132 bash[18607]: cluster 2024-04-25T20:19:59.158235+0000 mgr.x (mgr.14254) 777 : cluster [DBG] pgmap v631: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:01.575 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:20:02.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:01 smithi132 bash[18607]: audit 2024-04-25T20:19:59.871162+0000 mgr.x (mgr.14254) 778 : audit [DBG] from='client.14660 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:20:02.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:01 smithi132 bash[18607]: audit 2024-04-25T20:19:59.871162+0000 mgr.x (mgr.14254) 778 : audit [DBG] from='client.14660 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:20:03.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:02 smithi132 bash[18607]: cluster 2024-04-25T20:20:01.159660+0000 mgr.x (mgr.14254) 779 : cluster [DBG] pgmap v632: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:03.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:02 smithi132 bash[18607]: cluster 2024-04-25T20:20:01.159660+0000 mgr.x (mgr.14254) 779 : cluster [DBG] pgmap v632: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:05.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:04 smithi132 bash[18607]: cluster 2024-04-25T20:20:03.161474+0000 mgr.x (mgr.14254) 780 : cluster [DBG] pgmap v633: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:05.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:04 smithi132 bash[18607]: cluster 2024-04-25T20:20:03.161474+0000 mgr.x (mgr.14254) 780 : cluster [DBG] pgmap v633: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:06.335 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:20:07.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:06 smithi132 bash[18607]: cluster 2024-04-25T20:20:05.162800+0000 mgr.x (mgr.14254) 781 : cluster [DBG] pgmap v634: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:07.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:06 smithi132 bash[18607]: cluster 2024-04-25T20:20:05.162800+0000 mgr.x (mgr.14254) 781 : cluster [DBG] pgmap v634: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:07.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:06 smithi132 bash[18607]: audit 2024-04-25T20:20:05.924169+0000 mon.a (mon.0) 484 : audit [DBG] from='client.? 172.17.0.2:0/1673761115' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:20:07.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:06 smithi132 bash[18607]: audit 2024-04-25T20:20:05.924169+0000 mon.a (mon.0) 484 : audit [DBG] from='client.? 172.17.0.2:0/1673761115' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:20:08.884 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:08 smithi132 bash[18607]: cluster 2024-04-25T20:20:07.163869+0000 mgr.x (mgr.14254) 782 : cluster [DBG] pgmap v635: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:08.884 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:08 smithi132 bash[18607]: cluster 2024-04-25T20:20:07.163869+0000 mgr.x (mgr.14254) 782 : cluster [DBG] pgmap v635: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:09.272 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:20:09.272 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:20:09.920 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:20:10.921 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:20:10.933 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:10 smithi132 bash[18607]: cluster 2024-04-25T20:20:09.165633+0000 mgr.x (mgr.14254) 783 : cluster [DBG] pgmap v636: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:10.933 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:10 smithi132 bash[18607]: cluster 2024-04-25T20:20:09.165633+0000 mgr.x (mgr.14254) 783 : cluster [DBG] pgmap v636: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:10.933 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:10 smithi132 bash[18607]: audit 2024-04-25T20:20:09.275827+0000 mgr.x (mgr.14254) 784 : audit [DBG] from='client.14664 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:20:10.933 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:10 smithi132 bash[18607]: audit 2024-04-25T20:20:09.275827+0000 mgr.x (mgr.14254) 784 : audit [DBG] from='client.14664 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:20:13.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:12 smithi132 bash[18607]: cluster 2024-04-25T20:20:11.167038+0000 mgr.x (mgr.14254) 785 : cluster [DBG] pgmap v637: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:13.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:12 smithi132 bash[18607]: cluster 2024-04-25T20:20:11.167038+0000 mgr.x (mgr.14254) 785 : cluster [DBG] pgmap v637: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:15.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:14 smithi132 bash[18607]: cluster 2024-04-25T20:20:13.168478+0000 mgr.x (mgr.14254) 786 : cluster [DBG] pgmap v638: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:15.149 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:14 smithi132 bash[18607]: cluster 2024-04-25T20:20:13.168478+0000 mgr.x (mgr.14254) 786 : cluster [DBG] pgmap v638: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:15.682 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:20:16.971 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:16 smithi132 bash[18607]: cluster 2024-04-25T20:20:15.169857+0000 mgr.x (mgr.14254) 787 : cluster [DBG] pgmap v639: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:16.972 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:16 smithi132 bash[18607]: cluster 2024-04-25T20:20:15.169857+0000 mgr.x (mgr.14254) 787 : cluster [DBG] pgmap v639: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:18.542 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:20:18.542 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:20:19.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:18 smithi132 bash[18607]: cluster 2024-04-25T20:20:17.171277+0000 mgr.x (mgr.14254) 788 : cluster [DBG] pgmap v640: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:19.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:18 smithi132 bash[18607]: cluster 2024-04-25T20:20:17.171277+0000 mgr.x (mgr.14254) 788 : cluster [DBG] pgmap v640: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:19.223 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:20:20.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:19 smithi132 bash[18607]: audit 2024-04-25T20:20:18.544147+0000 mgr.x (mgr.14254) 789 : audit [DBG] from='client.14666 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:20:20.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:19 smithi132 bash[18607]: audit 2024-04-25T20:20:18.544147+0000 mgr.x (mgr.14254) 789 : audit [DBG] from='client.14666 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:20:20.224 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:20:21.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:20 smithi132 bash[18607]: cluster 2024-04-25T20:20:19.173286+0000 mgr.x (mgr.14254) 790 : cluster [DBG] pgmap v641: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:21.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:20 smithi132 bash[18607]: cluster 2024-04-25T20:20:19.173286+0000 mgr.x (mgr.14254) 790 : cluster [DBG] pgmap v641: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:22.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:21 smithi132 bash[18607]: audit 2024-04-25T20:20:21.263010+0000 mon.a (mon.0) 485 : audit [DBG] from='client.? 172.17.0.2:0/4031609743' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:20:22.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:21 smithi132 bash[18607]: audit 2024-04-25T20:20:21.263010+0000 mon.a (mon.0) 485 : audit [DBG] from='client.? 172.17.0.2:0/4031609743' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:20:23.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:22 smithi132 bash[18607]: cluster 2024-04-25T20:20:21.174753+0000 mgr.x (mgr.14254) 791 : cluster [DBG] pgmap v642: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:23.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:22 smithi132 bash[18607]: cluster 2024-04-25T20:20:21.174753+0000 mgr.x (mgr.14254) 791 : cluster [DBG] pgmap v642: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:24.982 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:20:25.003 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:24 smithi132 bash[18607]: cluster 2024-04-25T20:20:23.176666+0000 mgr.x (mgr.14254) 792 : cluster [DBG] pgmap v643: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:25.003 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:24 smithi132 bash[18607]: cluster 2024-04-25T20:20:23.176666+0000 mgr.x (mgr.14254) 792 : cluster [DBG] pgmap v643: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:27.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:26 smithi132 bash[18607]: cluster 2024-04-25T20:20:25.178064+0000 mgr.x (mgr.14254) 793 : cluster [DBG] pgmap v644: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:27.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:26 smithi132 bash[18607]: cluster 2024-04-25T20:20:25.178064+0000 mgr.x (mgr.14254) 793 : cluster [DBG] pgmap v644: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:27.921 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:20:27.921 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:20:28.553 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:20:29.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:28 smithi132 bash[18607]: cluster 2024-04-25T20:20:27.179483+0000 mgr.x (mgr.14254) 794 : cluster [DBG] pgmap v645: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:29.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:28 smithi132 bash[18607]: cluster 2024-04-25T20:20:27.179483+0000 mgr.x (mgr.14254) 794 : cluster [DBG] pgmap v645: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:29.555 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:20:30.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:29 smithi132 bash[18607]: audit 2024-04-25T20:20:27.922903+0000 mgr.x (mgr.14254) 795 : audit [DBG] from='client.14670 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:20:30.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:29 smithi132 bash[18607]: audit 2024-04-25T20:20:27.922903+0000 mgr.x (mgr.14254) 795 : audit [DBG] from='client.14670 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:20:31.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:30 smithi132 bash[18607]: cluster 2024-04-25T20:20:29.181365+0000 mgr.x (mgr.14254) 796 : cluster [DBG] pgmap v646: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:31.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:30 smithi132 bash[18607]: cluster 2024-04-25T20:20:29.181365+0000 mgr.x (mgr.14254) 796 : cluster [DBG] pgmap v646: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:33.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:32 smithi132 bash[18607]: cluster 2024-04-25T20:20:31.182779+0000 mgr.x (mgr.14254) 797 : cluster [DBG] pgmap v647: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:33.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:32 smithi132 bash[18607]: cluster 2024-04-25T20:20:31.182779+0000 mgr.x (mgr.14254) 797 : cluster [DBG] pgmap v647: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:34.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:33 smithi132 bash[18607]: cluster 2024-04-25T20:20:33.184580+0000 mgr.x (mgr.14254) 798 : cluster [DBG] pgmap v648: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:34.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:33 smithi132 bash[18607]: cluster 2024-04-25T20:20:33.184580+0000 mgr.x (mgr.14254) 798 : cluster [DBG] pgmap v648: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:34.316 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:20:36.519 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:36 smithi132 bash[18607]: cluster 2024-04-25T20:20:35.185927+0000 mgr.x (mgr.14254) 799 : cluster [DBG] pgmap v649: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:36.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:36 smithi132 bash[18607]: cluster 2024-04-25T20:20:35.185927+0000 mgr.x (mgr.14254) 799 : cluster [DBG] pgmap v649: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:37.174 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:20:37.174 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:20:37.519 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:37 smithi132 bash[18607]: audit 2024-04-25T20:20:36.597473+0000 mon.a (mon.0) 486 : audit [DBG] from='client.? 172.17.0.2:0/2978489256' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:20:37.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:37 smithi132 bash[18607]: audit 2024-04-25T20:20:36.597473+0000 mon.a (mon.0) 486 : audit [DBG] from='client.? 172.17.0.2:0/2978489256' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:20:37.924 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:20:38.519 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:38 smithi132 bash[18607]: audit 2024-04-25T20:20:37.175415+0000 mgr.x (mgr.14254) 800 : audit [DBG] from='client.14674 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:20:38.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:38 smithi132 bash[18607]: audit 2024-04-25T20:20:37.175415+0000 mgr.x (mgr.14254) 800 : audit [DBG] from='client.14674 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:20:38.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:38 smithi132 bash[18607]: cluster 2024-04-25T20:20:37.186841+0000 mgr.x (mgr.14254) 801 : cluster [DBG] pgmap v650: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:38.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:38 smithi132 bash[18607]: cluster 2024-04-25T20:20:37.186841+0000 mgr.x (mgr.14254) 801 : cluster [DBG] pgmap v650: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:38.925 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:20:41.019 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:40 smithi132 bash[18607]: cluster 2024-04-25T20:20:39.188727+0000 mgr.x (mgr.14254) 802 : cluster [DBG] pgmap v651: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:41.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:40 smithi132 bash[18607]: cluster 2024-04-25T20:20:39.188727+0000 mgr.x (mgr.14254) 802 : cluster [DBG] pgmap v651: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:43.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:42 smithi132 bash[18607]: cluster 2024-04-25T20:20:41.189583+0000 mgr.x (mgr.14254) 803 : cluster [DBG] pgmap v652: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:43.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:42 smithi132 bash[18607]: cluster 2024-04-25T20:20:41.189583+0000 mgr.x (mgr.14254) 803 : cluster [DBG] pgmap v652: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:43.682 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:20:44.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:44 smithi132 bash[18607]: cluster 2024-04-25T20:20:43.191281+0000 mgr.x (mgr.14254) 804 : cluster [DBG] pgmap v653: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:44.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:44 smithi132 bash[18607]: cluster 2024-04-25T20:20:43.191281+0000 mgr.x (mgr.14254) 804 : cluster [DBG] pgmap v653: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:46.438 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:20:46.438 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:20:46.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:46 smithi132 bash[18607]: cluster 2024-04-25T20:20:45.192734+0000 mgr.x (mgr.14254) 805 : cluster [DBG] pgmap v654: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:46.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:46 smithi132 bash[18607]: cluster 2024-04-25T20:20:45.192734+0000 mgr.x (mgr.14254) 805 : cluster [DBG] pgmap v654: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:47.135 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:20:48.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:47 smithi132 bash[18607]: audit 2024-04-25T20:20:46.439858+0000 mgr.x (mgr.14254) 806 : audit [DBG] from='client.14676 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:20:48.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:47 smithi132 bash[18607]: audit 2024-04-25T20:20:46.439858+0000 mgr.x (mgr.14254) 806 : audit [DBG] from='client.14676 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:20:48.136 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:20:49.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:48 smithi132 bash[18607]: cluster 2024-04-25T20:20:47.193968+0000 mgr.x (mgr.14254) 807 : cluster [DBG] pgmap v655: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:49.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:48 smithi132 bash[18607]: cluster 2024-04-25T20:20:47.193968+0000 mgr.x (mgr.14254) 807 : cluster [DBG] pgmap v655: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:51.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:50 smithi132 bash[18607]: cluster 2024-04-25T20:20:49.195835+0000 mgr.x (mgr.14254) 808 : cluster [DBG] pgmap v656: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:51.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:50 smithi132 bash[18607]: cluster 2024-04-25T20:20:49.195835+0000 mgr.x (mgr.14254) 808 : cluster [DBG] pgmap v656: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:52.897 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:20:52.919 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:52 smithi132 bash[18607]: cluster 2024-04-25T20:20:51.197325+0000 mgr.x (mgr.14254) 809 : cluster [DBG] pgmap v657: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:52.919 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:52 smithi132 bash[18607]: cluster 2024-04-25T20:20:51.197325+0000 mgr.x (mgr.14254) 809 : cluster [DBG] pgmap v657: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:52.919 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:52 smithi132 bash[18607]: audit 2024-04-25T20:20:51.934505+0000 mon.a (mon.0) 487 : audit [DBG] from='client.? 172.17.0.2:0/1027362504' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:20:52.919 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:52 smithi132 bash[18607]: audit 2024-04-25T20:20:51.934505+0000 mon.a (mon.0) 487 : audit [DBG] from='client.? 172.17.0.2:0/1027362504' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:20:55.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:54 smithi132 bash[18607]: cluster 2024-04-25T20:20:53.199073+0000 mgr.x (mgr.14254) 810 : cluster [DBG] pgmap v658: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:55.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:54 smithi132 bash[18607]: cluster 2024-04-25T20:20:53.199073+0000 mgr.x (mgr.14254) 810 : cluster [DBG] pgmap v658: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:55.745 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:20:55.745 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:20:56.431 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:20:56.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:56 smithi132 bash[18607]: cluster 2024-04-25T20:20:55.200468+0000 mgr.x (mgr.14254) 811 : cluster [DBG] pgmap v659: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:56.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:56 smithi132 bash[18607]: cluster 2024-04-25T20:20:55.200468+0000 mgr.x (mgr.14254) 811 : cluster [DBG] pgmap v659: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:57.432 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:20:57.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:57 smithi132 bash[18607]: audit 2024-04-25T20:20:55.747900+0000 mgr.x (mgr.14254) 812 : audit [DBG] from='client.14680 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:20:57.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:57 smithi132 bash[18607]: audit 2024-04-25T20:20:55.747900+0000 mgr.x (mgr.14254) 812 : audit [DBG] from='client.14680 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:20:59.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:58 smithi132 bash[18607]: cluster 2024-04-25T20:20:57.201860+0000 mgr.x (mgr.14254) 813 : cluster [DBG] pgmap v660: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:20:59.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:20:58 smithi132 bash[18607]: cluster 2024-04-25T20:20:57.201860+0000 mgr.x (mgr.14254) 813 : cluster [DBG] pgmap v660: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:01.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:00 smithi132 bash[18607]: cluster 2024-04-25T20:20:59.203707+0000 mgr.x (mgr.14254) 814 : cluster [DBG] pgmap v661: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:01.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:00 smithi132 bash[18607]: cluster 2024-04-25T20:20:59.203707+0000 mgr.x (mgr.14254) 814 : cluster [DBG] pgmap v661: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:02.190 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:21:03.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:02 smithi132 bash[18607]: cluster 2024-04-25T20:21:01.205157+0000 mgr.x (mgr.14254) 815 : cluster [DBG] pgmap v662: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:03.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:02 smithi132 bash[18607]: cluster 2024-04-25T20:21:01.205157+0000 mgr.x (mgr.14254) 815 : cluster [DBG] pgmap v662: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:05.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:04 smithi132 bash[18607]: cluster 2024-04-25T20:21:03.206926+0000 mgr.x (mgr.14254) 816 : cluster [DBG] pgmap v663: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:05.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:04 smithi132 bash[18607]: cluster 2024-04-25T20:21:03.206926+0000 mgr.x (mgr.14254) 816 : cluster [DBG] pgmap v663: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:05.188 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:21:05.188 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:21:05.824 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:21:06.825 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:21:07.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:06 smithi132 bash[18607]: audit 2024-04-25T20:21:05.187187+0000 mgr.x (mgr.14254) 817 : audit [DBG] from='client.14682 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:21:07.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:06 smithi132 bash[18607]: audit 2024-04-25T20:21:05.187187+0000 mgr.x (mgr.14254) 817 : audit [DBG] from='client.14682 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:21:07.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:06 smithi132 bash[18607]: cluster 2024-04-25T20:21:05.207792+0000 mgr.x (mgr.14254) 818 : cluster [DBG] pgmap v664: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:07.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:06 smithi132 bash[18607]: cluster 2024-04-25T20:21:05.207792+0000 mgr.x (mgr.14254) 818 : cluster [DBG] pgmap v664: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:08.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:07 smithi132 bash[18607]: audit 2024-04-25T20:21:07.272101+0000 mon.a (mon.0) 488 : audit [DBG] from='client.? 172.17.0.2:0/4069441067' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:21:08.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:07 smithi132 bash[18607]: audit 2024-04-25T20:21:07.272101+0000 mon.a (mon.0) 488 : audit [DBG] from='client.? 172.17.0.2:0/4069441067' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:21:09.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:08 smithi132 bash[18607]: cluster 2024-04-25T20:21:07.209316+0000 mgr.x (mgr.14254) 819 : cluster [DBG] pgmap v665: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:09.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:08 smithi132 bash[18607]: cluster 2024-04-25T20:21:07.209316+0000 mgr.x (mgr.14254) 819 : cluster [DBG] pgmap v665: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:11.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:10 smithi132 bash[18607]: cluster 2024-04-25T20:21:09.211331+0000 mgr.x (mgr.14254) 820 : cluster [DBG] pgmap v666: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:11.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:10 smithi132 bash[18607]: cluster 2024-04-25T20:21:09.211331+0000 mgr.x (mgr.14254) 820 : cluster [DBG] pgmap v666: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:11.586 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:21:12.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:12 smithi132 bash[18607]: cluster 2024-04-25T20:21:11.212753+0000 mgr.x (mgr.14254) 821 : cluster [DBG] pgmap v667: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:12.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:12 smithi132 bash[18607]: cluster 2024-04-25T20:21:11.212753+0000 mgr.x (mgr.14254) 821 : cluster [DBG] pgmap v667: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:14.335 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:21:14.335 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:21:15.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:14 smithi132 bash[18607]: cluster 2024-04-25T20:21:13.214494+0000 mgr.x (mgr.14254) 822 : cluster [DBG] pgmap v668: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:15.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:14 smithi132 bash[18607]: cluster 2024-04-25T20:21:13.214494+0000 mgr.x (mgr.14254) 822 : cluster [DBG] pgmap v668: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:15.092 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:21:16.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:15 smithi132 bash[18607]: audit 2024-04-25T20:21:14.337152+0000 mgr.x (mgr.14254) 823 : audit [DBG] from='client.14686 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:21:16.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:15 smithi132 bash[18607]: audit 2024-04-25T20:21:14.337152+0000 mgr.x (mgr.14254) 823 : audit [DBG] from='client.14686 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:21:16.094 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:21:17.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:16 smithi132 bash[18607]: cluster 2024-04-25T20:21:15.216019+0000 mgr.x (mgr.14254) 824 : cluster [DBG] pgmap v669: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:17.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:16 smithi132 bash[18607]: cluster 2024-04-25T20:21:15.216019+0000 mgr.x (mgr.14254) 824 : cluster [DBG] pgmap v669: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:19.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:18 smithi132 bash[18607]: cluster 2024-04-25T20:21:17.217480+0000 mgr.x (mgr.14254) 825 : cluster [DBG] pgmap v670: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:19.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:18 smithi132 bash[18607]: cluster 2024-04-25T20:21:17.217480+0000 mgr.x (mgr.14254) 825 : cluster [DBG] pgmap v670: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:20.854 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:21:20.876 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:20 smithi132 bash[18607]: cluster 2024-04-25T20:21:19.219507+0000 mgr.x (mgr.14254) 826 : cluster [DBG] pgmap v671: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:20.876 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:20 smithi132 bash[18607]: cluster 2024-04-25T20:21:19.219507+0000 mgr.x (mgr.14254) 826 : cluster [DBG] pgmap v671: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:23.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:22 smithi132 bash[18607]: cluster 2024-04-25T20:21:21.221033+0000 mgr.x (mgr.14254) 827 : cluster [DBG] pgmap v672: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:23.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:22 smithi132 bash[18607]: cluster 2024-04-25T20:21:21.221033+0000 mgr.x (mgr.14254) 827 : cluster [DBG] pgmap v672: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:23.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:22 smithi132 bash[18607]: audit 2024-04-25T20:21:22.605275+0000 mon.a (mon.0) 489 : audit [DBG] from='client.? 172.17.0.2:0/1194103998' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:21:23.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:22 smithi132 bash[18607]: audit 2024-04-25T20:21:22.605275+0000 mon.a (mon.0) 489 : audit [DBG] from='client.? 172.17.0.2:0/1194103998' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:21:23.689 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:21:23.689 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:21:24.434 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:21:24.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:24 smithi132 bash[18607]: cluster 2024-04-25T20:21:23.223034+0000 mgr.x (mgr.14254) 828 : cluster [DBG] pgmap v673: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:24.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:24 smithi132 bash[18607]: cluster 2024-04-25T20:21:23.223034+0000 mgr.x (mgr.14254) 828 : cluster [DBG] pgmap v673: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:25.436 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:21:25.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:25 smithi132 bash[18607]: audit 2024-04-25T20:21:23.687993+0000 mgr.x (mgr.14254) 829 : audit [DBG] from='client.14690 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:21:25.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:25 smithi132 bash[18607]: audit 2024-04-25T20:21:23.687993+0000 mgr.x (mgr.14254) 829 : audit [DBG] from='client.14690 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:21:27.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:26 smithi132 bash[18607]: cluster 2024-04-25T20:21:25.224493+0000 mgr.x (mgr.14254) 830 : cluster [DBG] pgmap v674: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:27.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:26 smithi132 bash[18607]: cluster 2024-04-25T20:21:25.224493+0000 mgr.x (mgr.14254) 830 : cluster [DBG] pgmap v674: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:29.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:28 smithi132 bash[18607]: cluster 2024-04-25T20:21:27.225917+0000 mgr.x (mgr.14254) 831 : cluster [DBG] pgmap v675: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:29.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:28 smithi132 bash[18607]: cluster 2024-04-25T20:21:27.225917+0000 mgr.x (mgr.14254) 831 : cluster [DBG] pgmap v675: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:30.190 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:21:31.009 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:30 smithi132 bash[18607]: cluster 2024-04-25T20:21:29.227788+0000 mgr.x (mgr.14254) 832 : cluster [DBG] pgmap v676: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:31.010 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:30 smithi132 bash[18607]: cluster 2024-04-25T20:21:29.227788+0000 mgr.x (mgr.14254) 832 : cluster [DBG] pgmap v676: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:32.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:32 smithi132 bash[18607]: cluster 2024-04-25T20:21:31.228764+0000 mgr.x (mgr.14254) 833 : cluster [DBG] pgmap v677: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:32.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:32 smithi132 bash[18607]: cluster 2024-04-25T20:21:31.228764+0000 mgr.x (mgr.14254) 833 : cluster [DBG] pgmap v677: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:32.920 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:21:32.920 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:21:33.680 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:21:34.682 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:21:35.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:34 smithi132 bash[18607]: audit 2024-04-25T20:21:32.922347+0000 mgr.x (mgr.14254) 834 : audit [DBG] from='client.14692 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:21:35.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:34 smithi132 bash[18607]: audit 2024-04-25T20:21:32.922347+0000 mgr.x (mgr.14254) 834 : audit [DBG] from='client.14692 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:21:35.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:34 smithi132 bash[18607]: cluster 2024-04-25T20:21:33.230562+0000 mgr.x (mgr.14254) 835 : cluster [DBG] pgmap v678: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:35.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:34 smithi132 bash[18607]: cluster 2024-04-25T20:21:33.230562+0000 mgr.x (mgr.14254) 835 : cluster [DBG] pgmap v678: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:37.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:36 smithi132 bash[18607]: cluster 2024-04-25T20:21:35.232043+0000 mgr.x (mgr.14254) 836 : cluster [DBG] pgmap v679: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:37.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:36 smithi132 bash[18607]: cluster 2024-04-25T20:21:35.232043+0000 mgr.x (mgr.14254) 836 : cluster [DBG] pgmap v679: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:39.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:38 smithi132 bash[18607]: cluster 2024-04-25T20:21:37.233480+0000 mgr.x (mgr.14254) 837 : cluster [DBG] pgmap v680: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:39.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:38 smithi132 bash[18607]: cluster 2024-04-25T20:21:37.233480+0000 mgr.x (mgr.14254) 837 : cluster [DBG] pgmap v680: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:39.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:38 smithi132 bash[18607]: audit 2024-04-25T20:21:37.940462+0000 mon.a (mon.0) 490 : audit [DBG] from='client.? 172.17.0.2:0/3380655610' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:21:39.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:38 smithi132 bash[18607]: audit 2024-04-25T20:21:37.940462+0000 mon.a (mon.0) 490 : audit [DBG] from='client.? 172.17.0.2:0/3380655610' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:21:39.434 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:21:41.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:40 smithi132 bash[18607]: cluster 2024-04-25T20:21:39.235375+0000 mgr.x (mgr.14254) 838 : cluster [DBG] pgmap v681: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:41.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:40 smithi132 bash[18607]: cluster 2024-04-25T20:21:39.235375+0000 mgr.x (mgr.14254) 838 : cluster [DBG] pgmap v681: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:42.249 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:21:42.249 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:21:42.908 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:42 smithi132 bash[18607]: cluster 2024-04-25T20:21:41.236810+0000 mgr.x (mgr.14254) 839 : cluster [DBG] pgmap v682: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:42.908 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:42 smithi132 bash[18607]: cluster 2024-04-25T20:21:41.236810+0000 mgr.x (mgr.14254) 839 : cluster [DBG] pgmap v682: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:42.909 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:21:43.911 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:21:43.921 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:43 smithi132 bash[18607]: audit 2024-04-25T20:21:42.247879+0000 mgr.x (mgr.14254) 840 : audit [DBG] from='client.14696 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:21:43.922 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:43 smithi132 bash[18607]: audit 2024-04-25T20:21:42.247879+0000 mgr.x (mgr.14254) 840 : audit [DBG] from='client.14696 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:21:45.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:44 smithi132 bash[18607]: cluster 2024-04-25T20:21:43.238720+0000 mgr.x (mgr.14254) 841 : cluster [DBG] pgmap v683: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:45.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:44 smithi132 bash[18607]: cluster 2024-04-25T20:21:43.238720+0000 mgr.x (mgr.14254) 841 : cluster [DBG] pgmap v683: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:47.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:46 smithi132 bash[18607]: cluster 2024-04-25T20:21:45.240146+0000 mgr.x (mgr.14254) 842 : cluster [DBG] pgmap v684: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:47.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:46 smithi132 bash[18607]: cluster 2024-04-25T20:21:45.240146+0000 mgr.x (mgr.14254) 842 : cluster [DBG] pgmap v684: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:48.666 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:21:49.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:48 smithi132 bash[18607]: cluster 2024-04-25T20:21:47.241593+0000 mgr.x (mgr.14254) 843 : cluster [DBG] pgmap v685: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:49.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:48 smithi132 bash[18607]: cluster 2024-04-25T20:21:47.241593+0000 mgr.x (mgr.14254) 843 : cluster [DBG] pgmap v685: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:51.008 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:50 smithi132 bash[18607]: cluster 2024-04-25T20:21:49.243460+0000 mgr.x (mgr.14254) 844 : cluster [DBG] pgmap v686: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:51.008 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:50 smithi132 bash[18607]: cluster 2024-04-25T20:21:49.243460+0000 mgr.x (mgr.14254) 844 : cluster [DBG] pgmap v686: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:51.400 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:21:51.400 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:21:52.050 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:21:53.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:52 smithi132 bash[18607]: cluster 2024-04-25T20:21:51.244828+0000 mgr.x (mgr.14254) 845 : cluster [DBG] pgmap v687: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:53.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:52 smithi132 bash[18607]: cluster 2024-04-25T20:21:51.244828+0000 mgr.x (mgr.14254) 845 : cluster [DBG] pgmap v687: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:53.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:52 smithi132 bash[18607]: audit 2024-04-25T20:21:51.398983+0000 mgr.x (mgr.14254) 846 : audit [DBG] from='client.14698 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:21:53.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:52 smithi132 bash[18607]: audit 2024-04-25T20:21:51.398983+0000 mgr.x (mgr.14254) 846 : audit [DBG] from='client.14698 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:21:53.051 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:21:54.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:53 smithi132 bash[18607]: audit 2024-04-25T20:21:53.266951+0000 mon.a (mon.0) 491 : audit [DBG] from='client.? 172.17.0.2:0/785050223' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:21:54.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:53 smithi132 bash[18607]: audit 2024-04-25T20:21:53.266951+0000 mon.a (mon.0) 491 : audit [DBG] from='client.? 172.17.0.2:0/785050223' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:21:55.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:54 smithi132 bash[18607]: cluster 2024-04-25T20:21:53.245663+0000 mgr.x (mgr.14254) 847 : cluster [DBG] pgmap v688: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:55.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:54 smithi132 bash[18607]: cluster 2024-04-25T20:21:53.245663+0000 mgr.x (mgr.14254) 847 : cluster [DBG] pgmap v688: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:57.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:56 smithi132 bash[18607]: cluster 2024-04-25T20:21:55.247019+0000 mgr.x (mgr.14254) 848 : cluster [DBG] pgmap v689: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:57.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:56 smithi132 bash[18607]: cluster 2024-04-25T20:21:55.247019+0000 mgr.x (mgr.14254) 848 : cluster [DBG] pgmap v689: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:57.810 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:21:58.997 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:58 smithi132 bash[18607]: cluster 2024-04-25T20:21:57.248435+0000 mgr.x (mgr.14254) 849 : cluster [DBG] pgmap v690: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:21:58.997 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:21:58 smithi132 bash[18607]: cluster 2024-04-25T20:21:57.248435+0000 mgr.x (mgr.14254) 849 : cluster [DBG] pgmap v690: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:00.593 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:22:00.593 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:22:01.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:00 smithi132 bash[18607]: cluster 2024-04-25T20:21:59.250371+0000 mgr.x (mgr.14254) 850 : cluster [DBG] pgmap v691: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:01.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:00 smithi132 bash[18607]: cluster 2024-04-25T20:21:59.250371+0000 mgr.x (mgr.14254) 850 : cluster [DBG] pgmap v691: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:01.299 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:22:02.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:01 smithi132 bash[18607]: audit 2024-04-25T20:22:00.592112+0000 mgr.x (mgr.14254) 851 : audit [DBG] from='client.14702 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:22:02.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:01 smithi132 bash[18607]: audit 2024-04-25T20:22:00.592112+0000 mgr.x (mgr.14254) 851 : audit [DBG] from='client.14702 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:22:02.300 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:22:03.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:02 smithi132 bash[18607]: cluster 2024-04-25T20:22:01.251797+0000 mgr.x (mgr.14254) 852 : cluster [DBG] pgmap v692: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:03.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:02 smithi132 bash[18607]: cluster 2024-04-25T20:22:01.251797+0000 mgr.x (mgr.14254) 852 : cluster [DBG] pgmap v692: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:04.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:03 smithi132 bash[18607]: cluster 2024-04-25T20:22:03.253618+0000 mgr.x (mgr.14254) 853 : cluster [DBG] pgmap v693: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:04.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:03 smithi132 bash[18607]: cluster 2024-04-25T20:22:03.253618+0000 mgr.x (mgr.14254) 853 : cluster [DBG] pgmap v693: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:06.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:06 smithi132 bash[18607]: cluster 2024-04-25T20:22:05.255048+0000 mgr.x (mgr.14254) 854 : cluster [DBG] pgmap v694: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:06.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:06 smithi132 bash[18607]: cluster 2024-04-25T20:22:05.255048+0000 mgr.x (mgr.14254) 854 : cluster [DBG] pgmap v694: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:07.059 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:22:08.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:08 smithi132 bash[18607]: cluster 2024-04-25T20:22:07.256441+0000 mgr.x (mgr.14254) 855 : cluster [DBG] pgmap v695: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:08.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:08 smithi132 bash[18607]: cluster 2024-04-25T20:22:07.256441+0000 mgr.x (mgr.14254) 855 : cluster [DBG] pgmap v695: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:09.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:09 smithi132 bash[18607]: audit 2024-04-25T20:22:08.592378+0000 mon.a (mon.0) 492 : audit [DBG] from='client.? 172.17.0.2:0/1847264615' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:22:09.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:09 smithi132 bash[18607]: audit 2024-04-25T20:22:08.592378+0000 mon.a (mon.0) 492 : audit [DBG] from='client.? 172.17.0.2:0/1847264615' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:22:09.899 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:22:09.899 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:22:10.535 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:22:10.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:10 smithi132 bash[18607]: cluster 2024-04-25T20:22:09.258480+0000 mgr.x (mgr.14254) 856 : cluster [DBG] pgmap v696: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:10.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:10 smithi132 bash[18607]: cluster 2024-04-25T20:22:09.258480+0000 mgr.x (mgr.14254) 856 : cluster [DBG] pgmap v696: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:11.536 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:22:11.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:11 smithi132 bash[18607]: audit 2024-04-25T20:22:09.900118+0000 mgr.x (mgr.14254) 857 : audit [DBG] from='client.14706 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:22:11.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:11 smithi132 bash[18607]: audit 2024-04-25T20:22:09.900118+0000 mgr.x (mgr.14254) 857 : audit [DBG] from='client.14706 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:22:12.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:12 smithi132 bash[18607]: cluster 2024-04-25T20:22:11.259949+0000 mgr.x (mgr.14254) 858 : cluster [DBG] pgmap v697: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:12.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:12 smithi132 bash[18607]: cluster 2024-04-25T20:22:11.259949+0000 mgr.x (mgr.14254) 858 : cluster [DBG] pgmap v697: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:14.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:14 smithi132 bash[18607]: cluster 2024-04-25T20:22:13.261980+0000 mgr.x (mgr.14254) 859 : cluster [DBG] pgmap v698: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:14.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:14 smithi132 bash[18607]: cluster 2024-04-25T20:22:13.261980+0000 mgr.x (mgr.14254) 859 : cluster [DBG] pgmap v698: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:16.294 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:22:16.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:16 smithi132 bash[18607]: cluster 2024-04-25T20:22:15.263460+0000 mgr.x (mgr.14254) 860 : cluster [DBG] pgmap v699: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:16.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:16 smithi132 bash[18607]: cluster 2024-04-25T20:22:15.263460+0000 mgr.x (mgr.14254) 860 : cluster [DBG] pgmap v699: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:18.645 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:18 smithi132 bash[18607]: cluster 2024-04-25T20:22:17.264230+0000 mgr.x (mgr.14254) 861 : cluster [DBG] pgmap v700: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:18.645 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:18 smithi132 bash[18607]: cluster 2024-04-25T20:22:17.264230+0000 mgr.x (mgr.14254) 861 : cluster [DBG] pgmap v700: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:19.099 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:22:19.099 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:22:19.863 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:22:20.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:20 smithi132 bash[18607]: audit 2024-04-25T20:22:19.097514+0000 mgr.x (mgr.14254) 862 : audit [DBG] from='client.14708 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:22:20.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:20 smithi132 bash[18607]: audit 2024-04-25T20:22:19.097514+0000 mgr.x (mgr.14254) 862 : audit [DBG] from='client.14708 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:22:20.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:20 smithi132 bash[18607]: cluster 2024-04-25T20:22:19.266144+0000 mgr.x (mgr.14254) 863 : cluster [DBG] pgmap v701: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:20.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:20 smithi132 bash[18607]: cluster 2024-04-25T20:22:19.266144+0000 mgr.x (mgr.14254) 863 : cluster [DBG] pgmap v701: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:20.864 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:22:22.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:22 smithi132 bash[18607]: cluster 2024-04-25T20:22:21.267611+0000 mgr.x (mgr.14254) 864 : cluster [DBG] pgmap v702: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:22.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:22 smithi132 bash[18607]: cluster 2024-04-25T20:22:21.267611+0000 mgr.x (mgr.14254) 864 : cluster [DBG] pgmap v702: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:24.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:24 smithi132 bash[18607]: cluster 2024-04-25T20:22:23.269467+0000 mgr.x (mgr.14254) 865 : cluster [DBG] pgmap v703: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:24.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:24 smithi132 bash[18607]: cluster 2024-04-25T20:22:23.269467+0000 mgr.x (mgr.14254) 865 : cluster [DBG] pgmap v703: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:24.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:24 smithi132 bash[18607]: audit 2024-04-25T20:22:23.927588+0000 mon.a (mon.0) 493 : audit [DBG] from='client.? 172.17.0.2:0/1278085006' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:22:24.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:24 smithi132 bash[18607]: audit 2024-04-25T20:22:23.927588+0000 mon.a (mon.0) 493 : audit [DBG] from='client.? 172.17.0.2:0/1278085006' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:22:25.626 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:22:26.594 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:26 smithi132 bash[18607]: cluster 2024-04-25T20:22:25.270872+0000 mgr.x (mgr.14254) 866 : cluster [DBG] pgmap v704: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:26.594 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:26 smithi132 bash[18607]: cluster 2024-04-25T20:22:25.270872+0000 mgr.x (mgr.14254) 866 : cluster [DBG] pgmap v704: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:28.403 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:22:28.403 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:22:28.626 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:28 smithi132 bash[18607]: cluster 2024-04-25T20:22:27.272216+0000 mgr.x (mgr.14254) 867 : cluster [DBG] pgmap v705: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:28.626 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:28 smithi132 bash[18607]: cluster 2024-04-25T20:22:27.272216+0000 mgr.x (mgr.14254) 867 : cluster [DBG] pgmap v705: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:29.112 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:22:29.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:29 smithi132 bash[18607]: audit 2024-04-25T20:22:28.404150+0000 mgr.x (mgr.14254) 868 : audit [DBG] from='client.14712 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:22:29.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:29 smithi132 bash[18607]: audit 2024-04-25T20:22:28.404150+0000 mgr.x (mgr.14254) 868 : audit [DBG] from='client.14712 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:22:30.113 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:22:30.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:30 smithi132 bash[18607]: cluster 2024-04-25T20:22:29.274137+0000 mgr.x (mgr.14254) 869 : cluster [DBG] pgmap v706: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:30.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:30 smithi132 bash[18607]: cluster 2024-04-25T20:22:29.274137+0000 mgr.x (mgr.14254) 869 : cluster [DBG] pgmap v706: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:32.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:32 smithi132 bash[18607]: cluster 2024-04-25T20:22:31.275601+0000 mgr.x (mgr.14254) 870 : cluster [DBG] pgmap v707: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:32.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:32 smithi132 bash[18607]: cluster 2024-04-25T20:22:31.275601+0000 mgr.x (mgr.14254) 870 : cluster [DBG] pgmap v707: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:34.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:34 smithi132 bash[18607]: cluster 2024-04-25T20:22:33.277406+0000 mgr.x (mgr.14254) 871 : cluster [DBG] pgmap v708: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:34.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:34 smithi132 bash[18607]: cluster 2024-04-25T20:22:33.277406+0000 mgr.x (mgr.14254) 871 : cluster [DBG] pgmap v708: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:34.868 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:22:36.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:36 smithi132 bash[18607]: cluster 2024-04-25T20:22:35.278767+0000 mgr.x (mgr.14254) 872 : cluster [DBG] pgmap v709: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:36.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:36 smithi132 bash[18607]: cluster 2024-04-25T20:22:35.278767+0000 mgr.x (mgr.14254) 872 : cluster [DBG] pgmap v709: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:37.655 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:22:37.655 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:22:38.306 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:22:38.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:38 smithi132 bash[18607]: cluster 2024-04-25T20:22:37.279636+0000 mgr.x (mgr.14254) 873 : cluster [DBG] pgmap v710: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:38.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:38 smithi132 bash[18607]: cluster 2024-04-25T20:22:37.279636+0000 mgr.x (mgr.14254) 873 : cluster [DBG] pgmap v710: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:38.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:38 smithi132 bash[18607]: audit 2024-04-25T20:22:37.653677+0000 mgr.x (mgr.14254) 874 : audit [DBG] from='client.14714 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:22:38.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:38 smithi132 bash[18607]: audit 2024-04-25T20:22:37.653677+0000 mgr.x (mgr.14254) 874 : audit [DBG] from='client.14714 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:22:39.307 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:22:39.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:39 smithi132 bash[18607]: audit 2024-04-25T20:22:39.262280+0000 mon.a (mon.0) 494 : audit [DBG] from='client.? 172.17.0.2:0/1390075308' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:22:39.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:39 smithi132 bash[18607]: audit 2024-04-25T20:22:39.262280+0000 mon.a (mon.0) 494 : audit [DBG] from='client.? 172.17.0.2:0/1390075308' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:22:40.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:40 smithi132 bash[18607]: cluster 2024-04-25T20:22:39.280615+0000 mgr.x (mgr.14254) 875 : cluster [DBG] pgmap v711: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:40.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:40 smithi132 bash[18607]: cluster 2024-04-25T20:22:39.280615+0000 mgr.x (mgr.14254) 875 : cluster [DBG] pgmap v711: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:42.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:42 smithi132 bash[18607]: cluster 2024-04-25T20:22:41.281979+0000 mgr.x (mgr.14254) 876 : cluster [DBG] pgmap v712: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:42.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:42 smithi132 bash[18607]: cluster 2024-04-25T20:22:41.281979+0000 mgr.x (mgr.14254) 876 : cluster [DBG] pgmap v712: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:44.048 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:22:44.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:44 smithi132 bash[18607]: cluster 2024-04-25T20:22:43.283764+0000 mgr.x (mgr.14254) 877 : cluster [DBG] pgmap v713: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:44.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:44 smithi132 bash[18607]: cluster 2024-04-25T20:22:43.283764+0000 mgr.x (mgr.14254) 877 : cluster [DBG] pgmap v713: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:46.616 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:46 smithi132 bash[18607]: cluster 2024-04-25T20:22:45.284742+0000 mgr.x (mgr.14254) 878 : cluster [DBG] pgmap v714: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:46.616 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:46 smithi132 bash[18607]: cluster 2024-04-25T20:22:45.284742+0000 mgr.x (mgr.14254) 878 : cluster [DBG] pgmap v714: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:47.003 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:22:47.004 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:22:47.675 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:22:48.676 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:22:48.687 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:48 smithi132 bash[18607]: audit 2024-04-25T20:22:47.004747+0000 mgr.x (mgr.14254) 879 : audit [DBG] from='client.14718 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:22:48.687 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:48 smithi132 bash[18607]: audit 2024-04-25T20:22:47.004747+0000 mgr.x (mgr.14254) 879 : audit [DBG] from='client.14718 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:22:48.687 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:48 smithi132 bash[18607]: cluster 2024-04-25T20:22:47.286061+0000 mgr.x (mgr.14254) 880 : cluster [DBG] pgmap v715: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:48.687 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:48 smithi132 bash[18607]: cluster 2024-04-25T20:22:47.286061+0000 mgr.x (mgr.14254) 880 : cluster [DBG] pgmap v715: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:50.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:50 smithi132 bash[18607]: cluster 2024-04-25T20:22:49.287943+0000 mgr.x (mgr.14254) 881 : cluster [DBG] pgmap v716: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:50.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:50 smithi132 bash[18607]: cluster 2024-04-25T20:22:49.287943+0000 mgr.x (mgr.14254) 881 : cluster [DBG] pgmap v716: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:52.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:52 smithi132 bash[18607]: cluster 2024-04-25T20:22:51.289315+0000 mgr.x (mgr.14254) 882 : cluster [DBG] pgmap v717: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:52.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:52 smithi132 bash[18607]: cluster 2024-04-25T20:22:51.289315+0000 mgr.x (mgr.14254) 882 : cluster [DBG] pgmap v717: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:53.436 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:22:54.621 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:54 smithi132 bash[18607]: cluster 2024-04-25T20:22:53.291168+0000 mgr.x (mgr.14254) 883 : cluster [DBG] pgmap v718: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:54.621 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:54 smithi132 bash[18607]: cluster 2024-04-25T20:22:53.291168+0000 mgr.x (mgr.14254) 883 : cluster [DBG] pgmap v718: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:55.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:55 smithi132 bash[18607]: audit 2024-04-25T20:22:54.592022+0000 mon.a (mon.0) 495 : audit [DBG] from='client.? 172.17.0.2:0/1540974187' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:22:55.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:55 smithi132 bash[18607]: audit 2024-04-25T20:22:54.592022+0000 mon.a (mon.0) 495 : audit [DBG] from='client.? 172.17.0.2:0/1540974187' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:22:56.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:56 smithi132 bash[18607]: cluster 2024-04-25T20:22:55.292338+0000 mgr.x (mgr.14254) 884 : cluster [DBG] pgmap v719: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:56.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:56 smithi132 bash[18607]: cluster 2024-04-25T20:22:55.292338+0000 mgr.x (mgr.14254) 884 : cluster [DBG] pgmap v719: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:56.616 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:22:56.616 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:22:57.269 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:22:57.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:57 smithi132 bash[18607]: audit 2024-04-25T20:22:56.614182+0000 mgr.x (mgr.14254) 885 : audit [DBG] from='client.14722 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:22:57.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:57 smithi132 bash[18607]: audit 2024-04-25T20:22:56.614182+0000 mgr.x (mgr.14254) 885 : audit [DBG] from='client.14722 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:22:58.270 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:22:58.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:58 smithi132 bash[18607]: cluster 2024-04-25T20:22:57.293818+0000 mgr.x (mgr.14254) 886 : cluster [DBG] pgmap v720: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:22:58.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:22:58 smithi132 bash[18607]: cluster 2024-04-25T20:22:57.293818+0000 mgr.x (mgr.14254) 886 : cluster [DBG] pgmap v720: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:00.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:00 smithi132 bash[18607]: cluster 2024-04-25T20:22:59.295708+0000 mgr.x (mgr.14254) 887 : cluster [DBG] pgmap v721: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:00.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:00 smithi132 bash[18607]: cluster 2024-04-25T20:22:59.295708+0000 mgr.x (mgr.14254) 887 : cluster [DBG] pgmap v721: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:02.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:02 smithi132 bash[18607]: cluster 2024-04-25T20:23:01.297292+0000 mgr.x (mgr.14254) 888 : cluster [DBG] pgmap v722: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:02.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:02 smithi132 bash[18607]: cluster 2024-04-25T20:23:01.297292+0000 mgr.x (mgr.14254) 888 : cluster [DBG] pgmap v722: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:03.025 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:23:04.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:04 smithi132 bash[18607]: cluster 2024-04-25T20:23:03.299086+0000 mgr.x (mgr.14254) 889 : cluster [DBG] pgmap v723: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:04.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:04 smithi132 bash[18607]: cluster 2024-04-25T20:23:03.299086+0000 mgr.x (mgr.14254) 889 : cluster [DBG] pgmap v723: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:05.824 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:23:05.824 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:23:06.518 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:23:06.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:06 smithi132 bash[18607]: cluster 2024-04-25T20:23:05.300489+0000 mgr.x (mgr.14254) 890 : cluster [DBG] pgmap v724: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:06.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:06 smithi132 bash[18607]: cluster 2024-04-25T20:23:05.300489+0000 mgr.x (mgr.14254) 890 : cluster [DBG] pgmap v724: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:07.520 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:23:07.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:07 smithi132 bash[18607]: audit 2024-04-25T20:23:05.822139+0000 mgr.x (mgr.14254) 891 : audit [DBG] from='client.14724 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:23:07.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:07 smithi132 bash[18607]: audit 2024-04-25T20:23:05.822139+0000 mgr.x (mgr.14254) 891 : audit [DBG] from='client.14724 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:23:08.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:08 smithi132 bash[18607]: cluster 2024-04-25T20:23:07.301886+0000 mgr.x (mgr.14254) 892 : cluster [DBG] pgmap v725: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:08.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:08 smithi132 bash[18607]: cluster 2024-04-25T20:23:07.301886+0000 mgr.x (mgr.14254) 892 : cluster [DBG] pgmap v725: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:10.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:10 smithi132 bash[18607]: cluster 2024-04-25T20:23:09.303739+0000 mgr.x (mgr.14254) 893 : cluster [DBG] pgmap v726: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:10.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:10 smithi132 bash[18607]: cluster 2024-04-25T20:23:09.303739+0000 mgr.x (mgr.14254) 893 : cluster [DBG] pgmap v726: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:10.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:10 smithi132 bash[18607]: audit 2024-04-25T20:23:09.920858+0000 mon.a (mon.0) 496 : audit [DBG] from='client.? 172.17.0.2:0/1770289798' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:23:10.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:10 smithi132 bash[18607]: audit 2024-04-25T20:23:09.920858+0000 mon.a (mon.0) 496 : audit [DBG] from='client.? 172.17.0.2:0/1770289798' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:23:12.273 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:23:12.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:12 smithi132 bash[18607]: cluster 2024-04-25T20:23:11.305191+0000 mgr.x (mgr.14254) 894 : cluster [DBG] pgmap v727: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:12.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:12 smithi132 bash[18607]: cluster 2024-04-25T20:23:11.305191+0000 mgr.x (mgr.14254) 894 : cluster [DBG] pgmap v727: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:14.730 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:14 smithi132 bash[18607]: cluster 2024-04-25T20:23:13.306150+0000 mgr.x (mgr.14254) 895 : cluster [DBG] pgmap v728: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:14.730 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:14 smithi132 bash[18607]: cluster 2024-04-25T20:23:13.306150+0000 mgr.x (mgr.14254) 895 : cluster [DBG] pgmap v728: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:15.147 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:23:15.147 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:23:15.886 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:23:16.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:16 smithi132 bash[18607]: audit 2024-04-25T20:23:15.147317+0000 mgr.x (mgr.14254) 896 : audit [DBG] from='client.14728 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:23:16.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:16 smithi132 bash[18607]: audit 2024-04-25T20:23:15.147317+0000 mgr.x (mgr.14254) 896 : audit [DBG] from='client.14728 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:23:16.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:16 smithi132 bash[18607]: cluster 2024-04-25T20:23:15.306909+0000 mgr.x (mgr.14254) 897 : cluster [DBG] pgmap v729: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:16.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:16 smithi132 bash[18607]: cluster 2024-04-25T20:23:15.306909+0000 mgr.x (mgr.14254) 897 : cluster [DBG] pgmap v729: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:16.887 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:23:18.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:18 smithi132 bash[18607]: cluster 2024-04-25T20:23:17.308356+0000 mgr.x (mgr.14254) 898 : cluster [DBG] pgmap v730: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:18.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:18 smithi132 bash[18607]: cluster 2024-04-25T20:23:17.308356+0000 mgr.x (mgr.14254) 898 : cluster [DBG] pgmap v730: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:20.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:20 smithi132 bash[18607]: cluster 2024-04-25T20:23:19.310169+0000 mgr.x (mgr.14254) 899 : cluster [DBG] pgmap v731: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:20.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:20 smithi132 bash[18607]: cluster 2024-04-25T20:23:19.310169+0000 mgr.x (mgr.14254) 899 : cluster [DBG] pgmap v731: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:21.648 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:23:22.710 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:22 smithi132 bash[18607]: cluster 2024-04-25T20:23:21.311585+0000 mgr.x (mgr.14254) 900 : cluster [DBG] pgmap v732: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:22.710 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:22 smithi132 bash[18607]: cluster 2024-04-25T20:23:21.311585+0000 mgr.x (mgr.14254) 900 : cluster [DBG] pgmap v732: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:24.449 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:23:24.492 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:23:24.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:24 smithi132 bash[18607]: cluster 2024-04-25T20:23:23.312538+0000 mgr.x (mgr.14254) 901 : cluster [DBG] pgmap v733: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:24.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:24 smithi132 bash[18607]: cluster 2024-04-25T20:23:23.312538+0000 mgr.x (mgr.14254) 901 : cluster [DBG] pgmap v733: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:25.129 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:23:25.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:25 smithi132 bash[18607]: audit 2024-04-25T20:23:24.449507+0000 mgr.x (mgr.14254) 902 : audit [DBG] from='client.14730 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:23:25.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:25 smithi132 bash[18607]: audit 2024-04-25T20:23:24.449507+0000 mgr.x (mgr.14254) 902 : audit [DBG] from='client.14730 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:23:25.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:25 smithi132 bash[18607]: audit 2024-04-25T20:23:25.251401+0000 mon.a (mon.0) 497 : audit [DBG] from='client.? 172.17.0.2:0/1171253822' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:23:25.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:25 smithi132 bash[18607]: audit 2024-04-25T20:23:25.251401+0000 mon.a (mon.0) 497 : audit [DBG] from='client.? 172.17.0.2:0/1171253822' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:23:26.130 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:23:26.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:26 smithi132 bash[18607]: cluster 2024-04-25T20:23:25.313887+0000 mgr.x (mgr.14254) 903 : cluster [DBG] pgmap v734: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:26.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:26 smithi132 bash[18607]: cluster 2024-04-25T20:23:25.313887+0000 mgr.x (mgr.14254) 903 : cluster [DBG] pgmap v734: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:28.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:28 smithi132 bash[18607]: cluster 2024-04-25T20:23:27.315315+0000 mgr.x (mgr.14254) 904 : cluster [DBG] pgmap v735: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:28.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:28 smithi132 bash[18607]: cluster 2024-04-25T20:23:27.315315+0000 mgr.x (mgr.14254) 904 : cluster [DBG] pgmap v735: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:30.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:30 smithi132 bash[18607]: cluster 2024-04-25T20:23:29.317294+0000 mgr.x (mgr.14254) 905 : cluster [DBG] pgmap v736: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:30.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:30 smithi132 bash[18607]: cluster 2024-04-25T20:23:29.317294+0000 mgr.x (mgr.14254) 905 : cluster [DBG] pgmap v736: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:30.886 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:23:32.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:32 smithi132 bash[18607]: cluster 2024-04-25T20:23:31.318720+0000 mgr.x (mgr.14254) 906 : cluster [DBG] pgmap v737: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:32.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:32 smithi132 bash[18607]: cluster 2024-04-25T20:23:31.318720+0000 mgr.x (mgr.14254) 906 : cluster [DBG] pgmap v737: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:33.750 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:23:33.750 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:23:34.452 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:23:34.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:34 smithi132 bash[18607]: cluster 2024-04-25T20:23:33.320664+0000 mgr.x (mgr.14254) 907 : cluster [DBG] pgmap v738: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:34.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:34 smithi132 bash[18607]: cluster 2024-04-25T20:23:33.320664+0000 mgr.x (mgr.14254) 907 : cluster [DBG] pgmap v738: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:35.453 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:23:35.557 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:35 smithi132 bash[18607]: audit 2024-04-25T20:23:33.750360+0000 mgr.x (mgr.14254) 908 : audit [DBG] from='client.14734 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:23:35.557 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:35 smithi132 bash[18607]: audit 2024-04-25T20:23:33.750360+0000 mgr.x (mgr.14254) 908 : audit [DBG] from='client.14734 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:23:36.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:36 smithi132 bash[18607]: cluster 2024-04-25T20:23:35.322051+0000 mgr.x (mgr.14254) 909 : cluster [DBG] pgmap v739: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:36.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:36 smithi132 bash[18607]: cluster 2024-04-25T20:23:35.322051+0000 mgr.x (mgr.14254) 909 : cluster [DBG] pgmap v739: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:38.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:38 smithi132 bash[18607]: cluster 2024-04-25T20:23:37.323433+0000 mgr.x (mgr.14254) 910 : cluster [DBG] pgmap v740: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:38.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:38 smithi132 bash[18607]: cluster 2024-04-25T20:23:37.323433+0000 mgr.x (mgr.14254) 910 : cluster [DBG] pgmap v740: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:40.215 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:23:40.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:40 smithi132 bash[18607]: cluster 2024-04-25T20:23:39.325322+0000 mgr.x (mgr.14254) 911 : cluster [DBG] pgmap v741: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:40.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:40 smithi132 bash[18607]: cluster 2024-04-25T20:23:39.325322+0000 mgr.x (mgr.14254) 911 : cluster [DBG] pgmap v741: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:41.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:41 smithi132 bash[18607]: audit 2024-04-25T20:23:40.586889+0000 mon.a (mon.0) 498 : audit [DBG] from='client.? 172.17.0.2:0/412120181' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:23:41.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:41 smithi132 bash[18607]: audit 2024-04-25T20:23:40.586889+0000 mon.a (mon.0) 498 : audit [DBG] from='client.? 172.17.0.2:0/412120181' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:23:42.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:42 smithi132 bash[18607]: cluster 2024-04-25T20:23:41.326043+0000 mgr.x (mgr.14254) 912 : cluster [DBG] pgmap v742: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:42.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:42 smithi132 bash[18607]: cluster 2024-04-25T20:23:41.326043+0000 mgr.x (mgr.14254) 912 : cluster [DBG] pgmap v742: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:43.187 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:23:43.188 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:23:43.896 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:23:44.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:44 smithi132 bash[18607]: audit 2024-04-25T20:23:43.185249+0000 mgr.x (mgr.14254) 913 : audit [DBG] from='client.14738 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:23:44.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:44 smithi132 bash[18607]: audit 2024-04-25T20:23:43.185249+0000 mgr.x (mgr.14254) 913 : audit [DBG] from='client.14738 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:23:44.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:44 smithi132 bash[18607]: cluster 2024-04-25T20:23:43.327821+0000 mgr.x (mgr.14254) 914 : cluster [DBG] pgmap v743: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:44.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:44 smithi132 bash[18607]: cluster 2024-04-25T20:23:43.327821+0000 mgr.x (mgr.14254) 914 : cluster [DBG] pgmap v743: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:44.897 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:23:46.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:46 smithi132 bash[18607]: cluster 2024-04-25T20:23:45.329270+0000 mgr.x (mgr.14254) 915 : cluster [DBG] pgmap v744: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:46.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:46 smithi132 bash[18607]: cluster 2024-04-25T20:23:45.329270+0000 mgr.x (mgr.14254) 915 : cluster [DBG] pgmap v744: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:48.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:48 smithi132 bash[18607]: cluster 2024-04-25T20:23:47.330801+0000 mgr.x (mgr.14254) 916 : cluster [DBG] pgmap v745: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:48.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:48 smithi132 bash[18607]: cluster 2024-04-25T20:23:47.330801+0000 mgr.x (mgr.14254) 916 : cluster [DBG] pgmap v745: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:49.663 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:23:50.714 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:50 smithi132 bash[18607]: cluster 2024-04-25T20:23:49.332856+0000 mgr.x (mgr.14254) 917 : cluster [DBG] pgmap v746: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:50.715 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:50 smithi132 bash[18607]: cluster 2024-04-25T20:23:49.332856+0000 mgr.x (mgr.14254) 917 : cluster [DBG] pgmap v746: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:52.516 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:23:52.516 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:23:52.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:52 smithi132 bash[18607]: cluster 2024-04-25T20:23:51.333720+0000 mgr.x (mgr.14254) 918 : cluster [DBG] pgmap v747: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:52.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:52 smithi132 bash[18607]: cluster 2024-04-25T20:23:51.333720+0000 mgr.x (mgr.14254) 918 : cluster [DBG] pgmap v747: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:53.164 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:23:53.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:53 smithi132 bash[18607]: audit 2024-04-25T20:23:52.515912+0000 mgr.x (mgr.14254) 919 : audit [DBG] from='client.14740 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:23:53.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:53 smithi132 bash[18607]: audit 2024-04-25T20:23:52.515912+0000 mgr.x (mgr.14254) 919 : audit [DBG] from='client.14740 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:23:54.165 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:23:54.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:54 smithi132 bash[18607]: cluster 2024-04-25T20:23:53.335579+0000 mgr.x (mgr.14254) 920 : cluster [DBG] pgmap v748: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:54.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:54 smithi132 bash[18607]: cluster 2024-04-25T20:23:53.335579+0000 mgr.x (mgr.14254) 920 : cluster [DBG] pgmap v748: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:56.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:56 smithi132 bash[18607]: cluster 2024-04-25T20:23:55.337141+0000 mgr.x (mgr.14254) 921 : cluster [DBG] pgmap v749: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:56.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:56 smithi132 bash[18607]: cluster 2024-04-25T20:23:55.337141+0000 mgr.x (mgr.14254) 921 : cluster [DBG] pgmap v749: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:58.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:58 smithi132 bash[18607]: cluster 2024-04-25T20:23:57.338695+0000 mgr.x (mgr.14254) 922 : cluster [DBG] pgmap v750: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:58.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:23:58 smithi132 bash[18607]: cluster 2024-04-25T20:23:57.338695+0000 mgr.x (mgr.14254) 922 : cluster [DBG] pgmap v750: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:23:58.922 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:24:00.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:00 smithi132 bash[18607]: cluster 2024-04-25T20:23:59.340561+0000 mgr.x (mgr.14254) 923 : cluster [DBG] pgmap v751: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:00.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:00 smithi132 bash[18607]: cluster 2024-04-25T20:23:59.340561+0000 mgr.x (mgr.14254) 923 : cluster [DBG] pgmap v751: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:01.907 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:24:01.907 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:24:02.620 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:24:02.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:02 smithi132 bash[18607]: cluster 2024-04-25T20:24:01.342089+0000 mgr.x (mgr.14254) 924 : cluster [DBG] pgmap v752: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:02.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:02 smithi132 bash[18607]: cluster 2024-04-25T20:24:01.342089+0000 mgr.x (mgr.14254) 924 : cluster [DBG] pgmap v752: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:03.622 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:24:03.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:03 smithi132 bash[18607]: audit 2024-04-25T20:24:01.907103+0000 mgr.x (mgr.14254) 925 : audit [DBG] from='client.14742 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:24:03.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:03 smithi132 bash[18607]: audit 2024-04-25T20:24:01.907103+0000 mgr.x (mgr.14254) 925 : audit [DBG] from='client.14742 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:24:04.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:04 smithi132 bash[18607]: cluster 2024-04-25T20:24:03.344071+0000 mgr.x (mgr.14254) 926 : cluster [DBG] pgmap v753: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:04.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:04 smithi132 bash[18607]: cluster 2024-04-25T20:24:03.344071+0000 mgr.x (mgr.14254) 926 : cluster [DBG] pgmap v753: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:06.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:06 smithi132 bash[18607]: cluster 2024-04-25T20:24:05.345637+0000 mgr.x (mgr.14254) 927 : cluster [DBG] pgmap v754: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:06.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:06 smithi132 bash[18607]: cluster 2024-04-25T20:24:05.345637+0000 mgr.x (mgr.14254) 927 : cluster [DBG] pgmap v754: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:08.385 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:24:08.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:08 smithi132 bash[18607]: cluster 2024-04-25T20:24:07.347203+0000 mgr.x (mgr.14254) 928 : cluster [DBG] pgmap v755: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:08.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:08 smithi132 bash[18607]: cluster 2024-04-25T20:24:07.347203+0000 mgr.x (mgr.14254) 928 : cluster [DBG] pgmap v755: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:10.735 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:10 smithi132 bash[18607]: cluster 2024-04-25T20:24:09.348270+0000 mgr.x (mgr.14254) 929 : cluster [DBG] pgmap v756: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:10.736 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:10 smithi132 bash[18607]: cluster 2024-04-25T20:24:09.348270+0000 mgr.x (mgr.14254) 929 : cluster [DBG] pgmap v756: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:11.130 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:24:11.130 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:24:11.888 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:24:12.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:12 smithi132 bash[18607]: audit 2024-04-25T20:24:11.130050+0000 mgr.x (mgr.14254) 930 : audit [DBG] from='client.14744 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:24:12.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:12 smithi132 bash[18607]: audit 2024-04-25T20:24:11.130050+0000 mgr.x (mgr.14254) 930 : audit [DBG] from='client.14744 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:24:12.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:12 smithi132 bash[18607]: cluster 2024-04-25T20:24:11.349701+0000 mgr.x (mgr.14254) 931 : cluster [DBG] pgmap v757: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:12.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:12 smithi132 bash[18607]: cluster 2024-04-25T20:24:11.349701+0000 mgr.x (mgr.14254) 931 : cluster [DBG] pgmap v757: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:12.889 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:24:14.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:14 smithi132 bash[18607]: cluster 2024-04-25T20:24:13.351498+0000 mgr.x (mgr.14254) 932 : cluster [DBG] pgmap v758: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:14.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:14 smithi132 bash[18607]: cluster 2024-04-25T20:24:13.351498+0000 mgr.x (mgr.14254) 932 : cluster [DBG] pgmap v758: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:16.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:16 smithi132 bash[18607]: cluster 2024-04-25T20:24:15.352923+0000 mgr.x (mgr.14254) 933 : cluster [DBG] pgmap v759: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:16.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:16 smithi132 bash[18607]: cluster 2024-04-25T20:24:15.352923+0000 mgr.x (mgr.14254) 933 : cluster [DBG] pgmap v759: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:17.651 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:24:18.726 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:18 smithi132 bash[18607]: cluster 2024-04-25T20:24:17.354379+0000 mgr.x (mgr.14254) 934 : cluster [DBG] pgmap v760: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:18.726 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:18 smithi132 bash[18607]: cluster 2024-04-25T20:24:17.354379+0000 mgr.x (mgr.14254) 934 : cluster [DBG] pgmap v760: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:20.272 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:24:20.272 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:24:20.774 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:20 smithi132 bash[18607]: cluster 2024-04-25T20:24:19.356195+0000 mgr.x (mgr.14254) 935 : cluster [DBG] pgmap v761: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:20.774 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:20 smithi132 bash[18607]: cluster 2024-04-25T20:24:19.356195+0000 mgr.x (mgr.14254) 935 : cluster [DBG] pgmap v761: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:20.967 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:24:21.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:21 smithi132 bash[18607]: audit 2024-04-25T20:24:20.271630+0000 mgr.x (mgr.14254) 936 : audit [DBG] from='client.14746 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:24:21.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:21 smithi132 bash[18607]: audit 2024-04-25T20:24:20.271630+0000 mgr.x (mgr.14254) 936 : audit [DBG] from='client.14746 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:24:21.968 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:24:22.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:22 smithi132 bash[18607]: cluster 2024-04-25T20:24:21.357802+0000 mgr.x (mgr.14254) 937 : cluster [DBG] pgmap v762: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:22.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:22 smithi132 bash[18607]: cluster 2024-04-25T20:24:21.357802+0000 mgr.x (mgr.14254) 937 : cluster [DBG] pgmap v762: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:24.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:24 smithi132 bash[18607]: cluster 2024-04-25T20:24:23.359678+0000 mgr.x (mgr.14254) 938 : cluster [DBG] pgmap v763: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:24.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:24 smithi132 bash[18607]: cluster 2024-04-25T20:24:23.359678+0000 mgr.x (mgr.14254) 938 : cluster [DBG] pgmap v763: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:26.725 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:24:26.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:26 smithi132 bash[18607]: cluster 2024-04-25T20:24:25.361110+0000 mgr.x (mgr.14254) 939 : cluster [DBG] pgmap v764: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:26.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:26 smithi132 bash[18607]: cluster 2024-04-25T20:24:25.361110+0000 mgr.x (mgr.14254) 939 : cluster [DBG] pgmap v764: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:28.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:28 smithi132 bash[18607]: cluster 2024-04-25T20:24:27.362587+0000 mgr.x (mgr.14254) 940 : cluster [DBG] pgmap v765: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:28.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:28 smithi132 bash[18607]: cluster 2024-04-25T20:24:27.362587+0000 mgr.x (mgr.14254) 940 : cluster [DBG] pgmap v765: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:29.412 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:24:29.412 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:24:30.172 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:24:30.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:30 smithi132 bash[18607]: cluster 2024-04-25T20:24:29.363685+0000 mgr.x (mgr.14254) 941 : cluster [DBG] pgmap v766: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:30.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:30 smithi132 bash[18607]: cluster 2024-04-25T20:24:29.363685+0000 mgr.x (mgr.14254) 941 : cluster [DBG] pgmap v766: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:30.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:30 smithi132 bash[18607]: audit 2024-04-25T20:24:29.411442+0000 mgr.x (mgr.14254) 942 : audit [DBG] from='client.14748 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:24:30.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:30 smithi132 bash[18607]: audit 2024-04-25T20:24:29.411442+0000 mgr.x (mgr.14254) 942 : audit [DBG] from='client.14748 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:24:31.173 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:24:32.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:32 smithi132 bash[18607]: cluster 2024-04-25T20:24:31.364314+0000 mgr.x (mgr.14254) 943 : cluster [DBG] pgmap v767: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:32.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:32 smithi132 bash[18607]: cluster 2024-04-25T20:24:31.364314+0000 mgr.x (mgr.14254) 943 : cluster [DBG] pgmap v767: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:34.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:34 smithi132 bash[18607]: cluster 2024-04-25T20:24:33.366112+0000 mgr.x (mgr.14254) 944 : cluster [DBG] pgmap v768: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:34.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:34 smithi132 bash[18607]: cluster 2024-04-25T20:24:33.366112+0000 mgr.x (mgr.14254) 944 : cluster [DBG] pgmap v768: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:35.926 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:24:36.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:36 smithi132 bash[18607]: cluster 2024-04-25T20:24:35.367513+0000 mgr.x (mgr.14254) 945 : cluster [DBG] pgmap v769: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:36.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:36 smithi132 bash[18607]: cluster 2024-04-25T20:24:35.367513+0000 mgr.x (mgr.14254) 945 : cluster [DBG] pgmap v769: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:38.576 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:24:38.577 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:24:38.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:38 smithi132 bash[18607]: cluster 2024-04-25T20:24:37.368983+0000 mgr.x (mgr.14254) 946 : cluster [DBG] pgmap v770: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:38.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:38 smithi132 bash[18607]: cluster 2024-04-25T20:24:37.368983+0000 mgr.x (mgr.14254) 946 : cluster [DBG] pgmap v770: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:39.310 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:24:40.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:39 smithi132 bash[18607]: audit 2024-04-25T20:24:38.576453+0000 mgr.x (mgr.14254) 947 : audit [DBG] from='client.14750 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:24:40.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:39 smithi132 bash[18607]: audit 2024-04-25T20:24:38.576453+0000 mgr.x (mgr.14254) 947 : audit [DBG] from='client.14750 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:24:40.311 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:24:41.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:40 smithi132 bash[18607]: cluster 2024-04-25T20:24:39.370952+0000 mgr.x (mgr.14254) 948 : cluster [DBG] pgmap v771: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:41.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:40 smithi132 bash[18607]: cluster 2024-04-25T20:24:39.370952+0000 mgr.x (mgr.14254) 948 : cluster [DBG] pgmap v771: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:43.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:42 smithi132 bash[18607]: cluster 2024-04-25T20:24:41.372391+0000 mgr.x (mgr.14254) 949 : cluster [DBG] pgmap v772: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:43.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:42 smithi132 bash[18607]: cluster 2024-04-25T20:24:41.372391+0000 mgr.x (mgr.14254) 949 : cluster [DBG] pgmap v772: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:45.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:44 smithi132 bash[18607]: cluster 2024-04-25T20:24:43.374226+0000 mgr.x (mgr.14254) 950 : cluster [DBG] pgmap v773: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:45.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:44 smithi132 bash[18607]: cluster 2024-04-25T20:24:43.374226+0000 mgr.x (mgr.14254) 950 : cluster [DBG] pgmap v773: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:45.068 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:24:47.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:46 smithi132 bash[18607]: cluster 2024-04-25T20:24:45.375621+0000 mgr.x (mgr.14254) 951 : cluster [DBG] pgmap v774: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:47.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:46 smithi132 bash[18607]: cluster 2024-04-25T20:24:45.375621+0000 mgr.x (mgr.14254) 951 : cluster [DBG] pgmap v774: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:47.975 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:24:47.975 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:24:48.621 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:24:49.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:48 smithi132 bash[18607]: cluster 2024-04-25T20:24:47.377162+0000 mgr.x (mgr.14254) 952 : cluster [DBG] pgmap v775: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:49.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:48 smithi132 bash[18607]: cluster 2024-04-25T20:24:47.377162+0000 mgr.x (mgr.14254) 952 : cluster [DBG] pgmap v775: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:49.623 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:24:50.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:49 smithi132 bash[18607]: audit 2024-04-25T20:24:47.975054+0000 mgr.x (mgr.14254) 953 : audit [DBG] from='client.14752 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:24:50.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:49 smithi132 bash[18607]: audit 2024-04-25T20:24:47.975054+0000 mgr.x (mgr.14254) 953 : audit [DBG] from='client.14752 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:24:51.013 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:50 smithi132 bash[18607]: cluster 2024-04-25T20:24:49.379061+0000 mgr.x (mgr.14254) 954 : cluster [DBG] pgmap v776: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:51.013 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:50 smithi132 bash[18607]: cluster 2024-04-25T20:24:49.379061+0000 mgr.x (mgr.14254) 954 : cluster [DBG] pgmap v776: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:52.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:51 smithi132 bash[18607]: audit 2024-04-25T20:24:50.971300+0000 mon.a (mon.0) 499 : audit [DBG] from='client.? 172.17.0.2:0/935694513' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:24:52.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:51 smithi132 bash[18607]: audit 2024-04-25T20:24:50.971300+0000 mon.a (mon.0) 499 : audit [DBG] from='client.? 172.17.0.2:0/935694513' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:24:53.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:52 smithi132 bash[18607]: cluster 2024-04-25T20:24:51.380494+0000 mgr.x (mgr.14254) 955 : cluster [DBG] pgmap v777: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:53.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:52 smithi132 bash[18607]: cluster 2024-04-25T20:24:51.380494+0000 mgr.x (mgr.14254) 955 : cluster [DBG] pgmap v777: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:54.383 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:24:54.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:54 smithi132 bash[18607]: cluster 2024-04-25T20:24:53.382299+0000 mgr.x (mgr.14254) 956 : cluster [DBG] pgmap v778: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:54.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:54 smithi132 bash[18607]: cluster 2024-04-25T20:24:53.382299+0000 mgr.x (mgr.14254) 956 : cluster [DBG] pgmap v778: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:56.916 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:56 smithi132 bash[18607]: cluster 2024-04-25T20:24:55.383296+0000 mgr.x (mgr.14254) 957 : cluster [DBG] pgmap v779: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:56.916 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:56 smithi132 bash[18607]: cluster 2024-04-25T20:24:55.383296+0000 mgr.x (mgr.14254) 957 : cluster [DBG] pgmap v779: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:57.301 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:24:57.301 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:24:58.003 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:24:59.004 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:24:59.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:58 smithi132 bash[18607]: audit 2024-04-25T20:24:57.301336+0000 mgr.x (mgr.14254) 958 : audit [DBG] from='client.14756 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:24:59.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:58 smithi132 bash[18607]: audit 2024-04-25T20:24:57.301336+0000 mgr.x (mgr.14254) 958 : audit [DBG] from='client.14756 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:24:59.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:58 smithi132 bash[18607]: cluster 2024-04-25T20:24:57.384318+0000 mgr.x (mgr.14254) 959 : cluster [DBG] pgmap v780: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:24:59.015 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:24:58 smithi132 bash[18607]: cluster 2024-04-25T20:24:57.384318+0000 mgr.x (mgr.14254) 959 : cluster [DBG] pgmap v780: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:01.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:00 smithi132 bash[18607]: cluster 2024-04-25T20:24:59.386213+0000 mgr.x (mgr.14254) 960 : cluster [DBG] pgmap v781: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:01.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:00 smithi132 bash[18607]: cluster 2024-04-25T20:24:59.386213+0000 mgr.x (mgr.14254) 960 : cluster [DBG] pgmap v781: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:03.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:02 smithi132 bash[18607]: cluster 2024-04-25T20:25:01.387798+0000 mgr.x (mgr.14254) 961 : cluster [DBG] pgmap v782: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:03.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:02 smithi132 bash[18607]: cluster 2024-04-25T20:25:01.387798+0000 mgr.x (mgr.14254) 961 : cluster [DBG] pgmap v782: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:03.755 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:25:04.943 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:04 smithi132 bash[18607]: cluster 2024-04-25T20:25:03.389966+0000 mgr.x (mgr.14254) 962 : cluster [DBG] pgmap v783: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:04.943 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:04 smithi132 bash[18607]: cluster 2024-04-25T20:25:03.389966+0000 mgr.x (mgr.14254) 962 : cluster [DBG] pgmap v783: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:06.582 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:25:06.582 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:25:07.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:06 smithi132 bash[18607]: cluster 2024-04-25T20:25:05.391578+0000 mgr.x (mgr.14254) 963 : cluster [DBG] pgmap v784: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:07.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:06 smithi132 bash[18607]: cluster 2024-04-25T20:25:05.391578+0000 mgr.x (mgr.14254) 963 : cluster [DBG] pgmap v784: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:07.320 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:25:08.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:07 smithi132 bash[18607]: audit 2024-04-25T20:25:06.579014+0000 mgr.x (mgr.14254) 964 : audit [DBG] from='client.14758 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:25:08.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:07 smithi132 bash[18607]: audit 2024-04-25T20:25:06.579014+0000 mgr.x (mgr.14254) 964 : audit [DBG] from='client.14758 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:25:08.321 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:25:09.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:08 smithi132 bash[18607]: cluster 2024-04-25T20:25:07.392789+0000 mgr.x (mgr.14254) 965 : cluster [DBG] pgmap v785: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:09.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:08 smithi132 bash[18607]: cluster 2024-04-25T20:25:07.392789+0000 mgr.x (mgr.14254) 965 : cluster [DBG] pgmap v785: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:11.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:10 smithi132 bash[18607]: cluster 2024-04-25T20:25:09.394764+0000 mgr.x (mgr.14254) 966 : cluster [DBG] pgmap v786: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:11.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:10 smithi132 bash[18607]: cluster 2024-04-25T20:25:09.394764+0000 mgr.x (mgr.14254) 966 : cluster [DBG] pgmap v786: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:13.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:12 smithi132 bash[18607]: cluster 2024-04-25T20:25:11.396338+0000 mgr.x (mgr.14254) 967 : cluster [DBG] pgmap v787: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:13.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:12 smithi132 bash[18607]: cluster 2024-04-25T20:25:11.396338+0000 mgr.x (mgr.14254) 967 : cluster [DBG] pgmap v787: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:13.074 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:25:15.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:14 smithi132 bash[18607]: cluster 2024-04-25T20:25:13.397571+0000 mgr.x (mgr.14254) 968 : cluster [DBG] pgmap v788: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:15.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:14 smithi132 bash[18607]: cluster 2024-04-25T20:25:13.397571+0000 mgr.x (mgr.14254) 968 : cluster [DBG] pgmap v788: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:15.892 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:25:15.892 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:25:16.591 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:25:17.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:16 smithi132 bash[18607]: cluster 2024-04-25T20:25:15.398990+0000 mgr.x (mgr.14254) 969 : cluster [DBG] pgmap v789: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:17.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:16 smithi132 bash[18607]: cluster 2024-04-25T20:25:15.398990+0000 mgr.x (mgr.14254) 969 : cluster [DBG] pgmap v789: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:17.592 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:25:18.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:17 smithi132 bash[18607]: audit 2024-04-25T20:25:15.889332+0000 mgr.x (mgr.14254) 970 : audit [DBG] from='client.14760 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:25:18.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:17 smithi132 bash[18607]: audit 2024-04-25T20:25:15.889332+0000 mgr.x (mgr.14254) 970 : audit [DBG] from='client.14760 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:25:19.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:18 smithi132 bash[18607]: cluster 2024-04-25T20:25:17.400376+0000 mgr.x (mgr.14254) 971 : cluster [DBG] pgmap v790: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:19.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:18 smithi132 bash[18607]: cluster 2024-04-25T20:25:17.400376+0000 mgr.x (mgr.14254) 971 : cluster [DBG] pgmap v790: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:21.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:20 smithi132 bash[18607]: cluster 2024-04-25T20:25:19.402253+0000 mgr.x (mgr.14254) 972 : cluster [DBG] pgmap v791: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:21.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:20 smithi132 bash[18607]: cluster 2024-04-25T20:25:19.402253+0000 mgr.x (mgr.14254) 972 : cluster [DBG] pgmap v791: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:22.350 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:25:23.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:22 smithi132 bash[18607]: cluster 2024-04-25T20:25:21.403704+0000 mgr.x (mgr.14254) 973 : cluster [DBG] pgmap v792: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:23.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:22 smithi132 bash[18607]: cluster 2024-04-25T20:25:21.403704+0000 mgr.x (mgr.14254) 973 : cluster [DBG] pgmap v792: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:25.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:24 smithi132 bash[18607]: cluster 2024-04-25T20:25:23.404804+0000 mgr.x (mgr.14254) 974 : cluster [DBG] pgmap v793: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:25.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:24 smithi132 bash[18607]: cluster 2024-04-25T20:25:23.404804+0000 mgr.x (mgr.14254) 974 : cluster [DBG] pgmap v793: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:25.221 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:25:25.221 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:25:25.958 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:25:26.960 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:25:26.971 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:26 smithi132 bash[18607]: audit 2024-04-25T20:25:25.218640+0000 mgr.x (mgr.14254) 975 : audit [DBG] from='client.14762 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:25:26.971 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:26 smithi132 bash[18607]: audit 2024-04-25T20:25:25.218640+0000 mgr.x (mgr.14254) 975 : audit [DBG] from='client.14762 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:25:26.971 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:26 smithi132 bash[18607]: cluster 2024-04-25T20:25:25.406356+0000 mgr.x (mgr.14254) 976 : cluster [DBG] pgmap v794: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:26.971 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:26 smithi132 bash[18607]: cluster 2024-04-25T20:25:25.406356+0000 mgr.x (mgr.14254) 976 : cluster [DBG] pgmap v794: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:29.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:28 smithi132 bash[18607]: cluster 2024-04-25T20:25:27.407931+0000 mgr.x (mgr.14254) 977 : cluster [DBG] pgmap v795: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:29.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:28 smithi132 bash[18607]: cluster 2024-04-25T20:25:27.407931+0000 mgr.x (mgr.14254) 977 : cluster [DBG] pgmap v795: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:31.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:30 smithi132 bash[18607]: cluster 2024-04-25T20:25:29.409881+0000 mgr.x (mgr.14254) 978 : cluster [DBG] pgmap v796: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:31.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:30 smithi132 bash[18607]: cluster 2024-04-25T20:25:29.409881+0000 mgr.x (mgr.14254) 978 : cluster [DBG] pgmap v796: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:31.718 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:25:32.922 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:32 smithi132 bash[18607]: cluster 2024-04-25T20:25:31.411299+0000 mgr.x (mgr.14254) 979 : cluster [DBG] pgmap v797: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:32.922 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:32 smithi132 bash[18607]: cluster 2024-04-25T20:25:31.411299+0000 mgr.x (mgr.14254) 979 : cluster [DBG] pgmap v797: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:34.494 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:25:34.495 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:25:34.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:34 smithi132 bash[18607]: cluster 2024-04-25T20:25:33.413202+0000 mgr.x (mgr.14254) 980 : cluster [DBG] pgmap v798: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:34.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:34 smithi132 bash[18607]: cluster 2024-04-25T20:25:33.413202+0000 mgr.x (mgr.14254) 980 : cluster [DBG] pgmap v798: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:35.271 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:25:36.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:35 smithi132 bash[18607]: audit 2024-04-25T20:25:34.491721+0000 mgr.x (mgr.14254) 981 : audit [DBG] from='client.14764 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:25:36.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:35 smithi132 bash[18607]: audit 2024-04-25T20:25:34.491721+0000 mgr.x (mgr.14254) 981 : audit [DBG] from='client.14764 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:25:36.272 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:25:37.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:36 smithi132 bash[18607]: cluster 2024-04-25T20:25:35.414637+0000 mgr.x (mgr.14254) 982 : cluster [DBG] pgmap v799: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:37.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:36 smithi132 bash[18607]: cluster 2024-04-25T20:25:35.414637+0000 mgr.x (mgr.14254) 982 : cluster [DBG] pgmap v799: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:39.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:38 smithi132 bash[18607]: cluster 2024-04-25T20:25:37.416076+0000 mgr.x (mgr.14254) 983 : cluster [DBG] pgmap v800: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:39.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:38 smithi132 bash[18607]: cluster 2024-04-25T20:25:37.416076+0000 mgr.x (mgr.14254) 983 : cluster [DBG] pgmap v800: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:41.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:40 smithi132 bash[18607]: cluster 2024-04-25T20:25:39.417948+0000 mgr.x (mgr.14254) 984 : cluster [DBG] pgmap v801: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:41.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:40 smithi132 bash[18607]: cluster 2024-04-25T20:25:39.417948+0000 mgr.x (mgr.14254) 984 : cluster [DBG] pgmap v801: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:41.031 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:25:43.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:42 smithi132 bash[18607]: cluster 2024-04-25T20:25:41.419384+0000 mgr.x (mgr.14254) 985 : cluster [DBG] pgmap v802: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:43.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:42 smithi132 bash[18607]: cluster 2024-04-25T20:25:41.419384+0000 mgr.x (mgr.14254) 985 : cluster [DBG] pgmap v802: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:43.875 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:25:43.875 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:25:44.592 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:25:45.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:44 smithi132 bash[18607]: cluster 2024-04-25T20:25:43.421277+0000 mgr.x (mgr.14254) 986 : cluster [DBG] pgmap v803: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:45.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:44 smithi132 bash[18607]: cluster 2024-04-25T20:25:43.421277+0000 mgr.x (mgr.14254) 986 : cluster [DBG] pgmap v803: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:45.594 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:25:46.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:45 smithi132 bash[18607]: audit 2024-04-25T20:25:43.874497+0000 mgr.x (mgr.14254) 987 : audit [DBG] from='client.14766 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:25:46.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:45 smithi132 bash[18607]: audit 2024-04-25T20:25:43.874497+0000 mgr.x (mgr.14254) 987 : audit [DBG] from='client.14766 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:25:47.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:46 smithi132 bash[18607]: cluster 2024-04-25T20:25:45.422690+0000 mgr.x (mgr.14254) 988 : cluster [DBG] pgmap v804: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:47.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:46 smithi132 bash[18607]: cluster 2024-04-25T20:25:45.422690+0000 mgr.x (mgr.14254) 988 : cluster [DBG] pgmap v804: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:49.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:48 smithi132 bash[18607]: cluster 2024-04-25T20:25:47.424138+0000 mgr.x (mgr.14254) 989 : cluster [DBG] pgmap v805: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:49.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:48 smithi132 bash[18607]: cluster 2024-04-25T20:25:47.424138+0000 mgr.x (mgr.14254) 989 : cluster [DBG] pgmap v805: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:50.353 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:25:51.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:50 smithi132 bash[18607]: cluster 2024-04-25T20:25:49.426057+0000 mgr.x (mgr.14254) 990 : cluster [DBG] pgmap v806: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:51.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:50 smithi132 bash[18607]: cluster 2024-04-25T20:25:49.426057+0000 mgr.x (mgr.14254) 990 : cluster [DBG] pgmap v806: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:53.000 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:25:53.000 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:25:53.014 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:52 smithi132 bash[18607]: cluster 2024-04-25T20:25:51.427553+0000 mgr.x (mgr.14254) 991 : cluster [DBG] pgmap v807: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:53.014 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:52 smithi132 bash[18607]: cluster 2024-04-25T20:25:51.427553+0000 mgr.x (mgr.14254) 991 : cluster [DBG] pgmap v807: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:53.714 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:25:54.715 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:25:55.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:54 smithi132 bash[18607]: audit 2024-04-25T20:25:52.997527+0000 mgr.x (mgr.14254) 992 : audit [DBG] from='client.14768 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:25:55.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:54 smithi132 bash[18607]: audit 2024-04-25T20:25:52.997527+0000 mgr.x (mgr.14254) 992 : audit [DBG] from='client.14768 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:25:55.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:54 smithi132 bash[18607]: cluster 2024-04-25T20:25:53.429585+0000 mgr.x (mgr.14254) 993 : cluster [DBG] pgmap v808: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:55.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:54 smithi132 bash[18607]: cluster 2024-04-25T20:25:53.429585+0000 mgr.x (mgr.14254) 993 : cluster [DBG] pgmap v808: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:57.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:56 smithi132 bash[18607]: cluster 2024-04-25T20:25:55.431070+0000 mgr.x (mgr.14254) 994 : cluster [DBG] pgmap v809: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:57.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:56 smithi132 bash[18607]: cluster 2024-04-25T20:25:55.431070+0000 mgr.x (mgr.14254) 994 : cluster [DBG] pgmap v809: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:59.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:58 smithi132 bash[18607]: cluster 2024-04-25T20:25:57.432457+0000 mgr.x (mgr.14254) 995 : cluster [DBG] pgmap v810: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:59.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:58 smithi132 bash[18607]: cluster 2024-04-25T20:25:57.432457+0000 mgr.x (mgr.14254) 995 : cluster [DBG] pgmap v810: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:59.474 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:25:59.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:59 smithi132 bash[18607]: cluster 2024-04-25T20:25:59.434498+0000 mgr.x (mgr.14254) 996 : cluster [DBG] pgmap v811: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:25:59.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:25:59 smithi132 bash[18607]: cluster 2024-04-25T20:25:59.434498+0000 mgr.x (mgr.14254) 996 : cluster [DBG] pgmap v811: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:01.522 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:01 smithi132 bash[18607]: audit 2024-04-25T20:26:01.352718+0000 mon.a (mon.0) 500 : audit [DBG] from='client.? 172.17.0.2:0/4293205439' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:26:01.522 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:01 smithi132 bash[18607]: audit 2024-04-25T20:26:01.352718+0000 mon.a (mon.0) 500 : audit [DBG] from='client.? 172.17.0.2:0/4293205439' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:26:02.320 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:26:02.320 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:26:02.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:02 smithi132 bash[18607]: cluster 2024-04-25T20:26:01.435983+0000 mgr.x (mgr.14254) 997 : cluster [DBG] pgmap v812: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:02.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:02 smithi132 bash[18607]: cluster 2024-04-25T20:26:01.435983+0000 mgr.x (mgr.14254) 997 : cluster [DBG] pgmap v812: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:03.059 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:26:03.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:03 smithi132 bash[18607]: audit 2024-04-25T20:26:02.317274+0000 mgr.x (mgr.14254) 998 : audit [DBG] from='client.14772 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:26:03.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:03 smithi132 bash[18607]: audit 2024-04-25T20:26:02.317274+0000 mgr.x (mgr.14254) 998 : audit [DBG] from='client.14772 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:26:04.060 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:26:04.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:04 smithi132 bash[18607]: cluster 2024-04-25T20:26:03.437887+0000 mgr.x (mgr.14254) 999 : cluster [DBG] pgmap v813: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:04.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:04 smithi132 bash[18607]: cluster 2024-04-25T20:26:03.437887+0000 mgr.x (mgr.14254) 999 : cluster [DBG] pgmap v813: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:06.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:06 smithi132 bash[18607]: cluster 2024-04-25T20:26:05.439304+0000 mgr.x (mgr.14254) 1000 : cluster [DBG] pgmap v814: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:06.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:06 smithi132 bash[18607]: cluster 2024-04-25T20:26:05.439304+0000 mgr.x (mgr.14254) 1000 : cluster [DBG] pgmap v814: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:08.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:08 smithi132 bash[18607]: cluster 2024-04-25T20:26:07.440868+0000 mgr.x (mgr.14254) 1001 : cluster [DBG] pgmap v815: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:08.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:08 smithi132 bash[18607]: cluster 2024-04-25T20:26:07.440868+0000 mgr.x (mgr.14254) 1001 : cluster [DBG] pgmap v815: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:08.819 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:26:10.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:10 smithi132 bash[18607]: cluster 2024-04-25T20:26:09.442258+0000 mgr.x (mgr.14254) 1002 : cluster [DBG] pgmap v816: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:10.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:10 smithi132 bash[18607]: cluster 2024-04-25T20:26:09.442258+0000 mgr.x (mgr.14254) 1002 : cluster [DBG] pgmap v816: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:11.550 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:26:11.550 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:26:12.293 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:26:12.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:12 smithi132 bash[18607]: cluster 2024-04-25T20:26:11.443668+0000 mgr.x (mgr.14254) 1003 : cluster [DBG] pgmap v817: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:12.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:12 smithi132 bash[18607]: cluster 2024-04-25T20:26:11.443668+0000 mgr.x (mgr.14254) 1003 : cluster [DBG] pgmap v817: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:12.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:12 smithi132 bash[18607]: audit 2024-04-25T20:26:11.547346+0000 mgr.x (mgr.14254) 1004 : audit [DBG] from='client.14774 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:26:12.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:12 smithi132 bash[18607]: audit 2024-04-25T20:26:11.547346+0000 mgr.x (mgr.14254) 1004 : audit [DBG] from='client.14774 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:26:13.294 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:26:14.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:14 smithi132 bash[18607]: cluster 2024-04-25T20:26:13.445522+0000 mgr.x (mgr.14254) 1005 : cluster [DBG] pgmap v818: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:14.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:14 smithi132 bash[18607]: cluster 2024-04-25T20:26:13.445522+0000 mgr.x (mgr.14254) 1005 : cluster [DBG] pgmap v818: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:16.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:16 smithi132 bash[18607]: cluster 2024-04-25T20:26:15.446885+0000 mgr.x (mgr.14254) 1006 : cluster [DBG] pgmap v819: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:16.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:16 smithi132 bash[18607]: cluster 2024-04-25T20:26:15.446885+0000 mgr.x (mgr.14254) 1006 : cluster [DBG] pgmap v819: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:18.051 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:26:18.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:18 smithi132 bash[18607]: cluster 2024-04-25T20:26:17.448331+0000 mgr.x (mgr.14254) 1007 : cluster [DBG] pgmap v820: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:18.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:18 smithi132 bash[18607]: cluster 2024-04-25T20:26:17.448331+0000 mgr.x (mgr.14254) 1007 : cluster [DBG] pgmap v820: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:20.743 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:26:20.743 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:26:20.756 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:20 smithi132 bash[18607]: cluster 2024-04-25T20:26:19.450355+0000 mgr.x (mgr.14254) 1008 : cluster [DBG] pgmap v821: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:20.756 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:20 smithi132 bash[18607]: cluster 2024-04-25T20:26:19.450355+0000 mgr.x (mgr.14254) 1008 : cluster [DBG] pgmap v821: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:21.396 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:26:22.397 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:26:22.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:22 smithi132 bash[18607]: audit 2024-04-25T20:26:20.740287+0000 mgr.x (mgr.14254) 1009 : audit [DBG] from='client.14776 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:26:22.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:22 smithi132 bash[18607]: audit 2024-04-25T20:26:20.740287+0000 mgr.x (mgr.14254) 1009 : audit [DBG] from='client.14776 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:26:22.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:22 smithi132 bash[18607]: cluster 2024-04-25T20:26:21.451866+0000 mgr.x (mgr.14254) 1010 : cluster [DBG] pgmap v822: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:22.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:22 smithi132 bash[18607]: cluster 2024-04-25T20:26:21.451866+0000 mgr.x (mgr.14254) 1010 : cluster [DBG] pgmap v822: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:24.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:24 smithi132 bash[18607]: cluster 2024-04-25T20:26:23.453782+0000 mgr.x (mgr.14254) 1011 : cluster [DBG] pgmap v823: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:24.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:24 smithi132 bash[18607]: cluster 2024-04-25T20:26:23.453782+0000 mgr.x (mgr.14254) 1011 : cluster [DBG] pgmap v823: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:26.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:26 smithi132 bash[18607]: cluster 2024-04-25T20:26:25.455396+0000 mgr.x (mgr.14254) 1012 : cluster [DBG] pgmap v824: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:26.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:26 smithi132 bash[18607]: cluster 2024-04-25T20:26:25.455396+0000 mgr.x (mgr.14254) 1012 : cluster [DBG] pgmap v824: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:27.159 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:26:28.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:28 smithi132 bash[18607]: cluster 2024-04-25T20:26:27.457011+0000 mgr.x (mgr.14254) 1013 : cluster [DBG] pgmap v825: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:28.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:28 smithi132 bash[18607]: cluster 2024-04-25T20:26:27.457011+0000 mgr.x (mgr.14254) 1013 : cluster [DBG] pgmap v825: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:29.915 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:26:29.915 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:26:30.621 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:26:30.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:30 smithi132 bash[18607]: cluster 2024-04-25T20:26:29.458943+0000 mgr.x (mgr.14254) 1014 : cluster [DBG] pgmap v826: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:30.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:30 smithi132 bash[18607]: cluster 2024-04-25T20:26:29.458943+0000 mgr.x (mgr.14254) 1014 : cluster [DBG] pgmap v826: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:31.623 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:26:31.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:31 smithi132 bash[18607]: audit 2024-04-25T20:26:29.914587+0000 mgr.x (mgr.14254) 1015 : audit [DBG] from='client.14778 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:26:31.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:31 smithi132 bash[18607]: audit 2024-04-25T20:26:29.914587+0000 mgr.x (mgr.14254) 1015 : audit [DBG] from='client.14778 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:26:32.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:32 smithi132 bash[18607]: cluster 2024-04-25T20:26:31.460388+0000 mgr.x (mgr.14254) 1016 : cluster [DBG] pgmap v827: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:32.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:32 smithi132 bash[18607]: cluster 2024-04-25T20:26:31.460388+0000 mgr.x (mgr.14254) 1016 : cluster [DBG] pgmap v827: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:35.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:34 smithi132 bash[18607]: cluster 2024-04-25T20:26:33.462184+0000 mgr.x (mgr.14254) 1017 : cluster [DBG] pgmap v828: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:35.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:34 smithi132 bash[18607]: cluster 2024-04-25T20:26:33.462184+0000 mgr.x (mgr.14254) 1017 : cluster [DBG] pgmap v828: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:36.384 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:26:36.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:36 smithi132 bash[18607]: cluster 2024-04-25T20:26:35.463582+0000 mgr.x (mgr.14254) 1018 : cluster [DBG] pgmap v829: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:36.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:36 smithi132 bash[18607]: cluster 2024-04-25T20:26:35.463582+0000 mgr.x (mgr.14254) 1018 : cluster [DBG] pgmap v829: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:38.784 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:38 smithi132 bash[18607]: cluster 2024-04-25T20:26:37.464431+0000 mgr.x (mgr.14254) 1019 : cluster [DBG] pgmap v830: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:38.784 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:38 smithi132 bash[18607]: cluster 2024-04-25T20:26:37.464431+0000 mgr.x (mgr.14254) 1019 : cluster [DBG] pgmap v830: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:39.147 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:26:39.147 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:26:39.853 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:26:40.854 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:26:40.865 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:40 smithi132 bash[18607]: audit 2024-04-25T20:26:39.147471+0000 mgr.x (mgr.14254) 1020 : audit [DBG] from='client.14780 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:26:40.865 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:40 smithi132 bash[18607]: audit 2024-04-25T20:26:39.147471+0000 mgr.x (mgr.14254) 1020 : audit [DBG] from='client.14780 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:26:40.865 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:40 smithi132 bash[18607]: cluster 2024-04-25T20:26:39.466525+0000 mgr.x (mgr.14254) 1021 : cluster [DBG] pgmap v831: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:40.865 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:40 smithi132 bash[18607]: cluster 2024-04-25T20:26:39.466525+0000 mgr.x (mgr.14254) 1021 : cluster [DBG] pgmap v831: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:43.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:42 smithi132 bash[18607]: cluster 2024-04-25T20:26:41.467941+0000 mgr.x (mgr.14254) 1022 : cluster [DBG] pgmap v832: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:43.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:42 smithi132 bash[18607]: cluster 2024-04-25T20:26:41.467941+0000 mgr.x (mgr.14254) 1022 : cluster [DBG] pgmap v832: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:45.026 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:44 smithi132 bash[18607]: cluster 2024-04-25T20:26:43.469804+0000 mgr.x (mgr.14254) 1023 : cluster [DBG] pgmap v833: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:45.027 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:44 smithi132 bash[18607]: cluster 2024-04-25T20:26:43.469804+0000 mgr.x (mgr.14254) 1023 : cluster [DBG] pgmap v833: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:45.612 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:26:46.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:46 smithi132 bash[18607]: cluster 2024-04-25T20:26:45.471284+0000 mgr.x (mgr.14254) 1024 : cluster [DBG] pgmap v834: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:46.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:46 smithi132 bash[18607]: cluster 2024-04-25T20:26:45.471284+0000 mgr.x (mgr.14254) 1024 : cluster [DBG] pgmap v834: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:48.416 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:26:48.416 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:26:48.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:48 smithi132 bash[18607]: cluster 2024-04-25T20:26:47.472813+0000 mgr.x (mgr.14254) 1025 : cluster [DBG] pgmap v835: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:48.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:48 smithi132 bash[18607]: cluster 2024-04-25T20:26:47.472813+0000 mgr.x (mgr.14254) 1025 : cluster [DBG] pgmap v835: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:49.060 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:26:50.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:49 smithi132 bash[18607]: audit 2024-04-25T20:26:48.412873+0000 mgr.x (mgr.14254) 1026 : audit [DBG] from='client.14782 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:26:50.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:49 smithi132 bash[18607]: audit 2024-04-25T20:26:48.412873+0000 mgr.x (mgr.14254) 1026 : audit [DBG] from='client.14782 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:26:50.062 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:26:51.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:50 smithi132 bash[18607]: cluster 2024-04-25T20:26:49.474756+0000 mgr.x (mgr.14254) 1027 : cluster [DBG] pgmap v836: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:51.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:50 smithi132 bash[18607]: cluster 2024-04-25T20:26:49.474756+0000 mgr.x (mgr.14254) 1027 : cluster [DBG] pgmap v836: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:53.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:52 smithi132 bash[18607]: cluster 2024-04-25T20:26:51.476177+0000 mgr.x (mgr.14254) 1028 : cluster [DBG] pgmap v837: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:53.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:52 smithi132 bash[18607]: cluster 2024-04-25T20:26:51.476177+0000 mgr.x (mgr.14254) 1028 : cluster [DBG] pgmap v837: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:54.818 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:26:54.841 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:54 smithi132 bash[18607]: cluster 2024-04-25T20:26:53.477982+0000 mgr.x (mgr.14254) 1029 : cluster [DBG] pgmap v838: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:54.841 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:54 smithi132 bash[18607]: cluster 2024-04-25T20:26:53.477982+0000 mgr.x (mgr.14254) 1029 : cluster [DBG] pgmap v838: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:56.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:56 smithi132 bash[18607]: cluster 2024-04-25T20:26:55.479454+0000 mgr.x (mgr.14254) 1030 : cluster [DBG] pgmap v839: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:56.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:56 smithi132 bash[18607]: cluster 2024-04-25T20:26:55.479454+0000 mgr.x (mgr.14254) 1030 : cluster [DBG] pgmap v839: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:57.432 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:26:57.432 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:26:58.104 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:26:59.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:58 smithi132 bash[18607]: audit 2024-04-25T20:26:57.432895+0000 mgr.x (mgr.14254) 1031 : audit [DBG] from='client.14784 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:26:59.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:58 smithi132 bash[18607]: audit 2024-04-25T20:26:57.432895+0000 mgr.x (mgr.14254) 1031 : audit [DBG] from='client.14784 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:26:59.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:58 smithi132 bash[18607]: cluster 2024-04-25T20:26:57.480227+0000 mgr.x (mgr.14254) 1032 : cluster [DBG] pgmap v840: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:59.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:26:58 smithi132 bash[18607]: cluster 2024-04-25T20:26:57.480227+0000 mgr.x (mgr.14254) 1032 : cluster [DBG] pgmap v840: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:26:59.106 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:27:01.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:00 smithi132 bash[18607]: cluster 2024-04-25T20:26:59.482218+0000 mgr.x (mgr.14254) 1033 : cluster [DBG] pgmap v841: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:01.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:00 smithi132 bash[18607]: cluster 2024-04-25T20:26:59.482218+0000 mgr.x (mgr.14254) 1033 : cluster [DBG] pgmap v841: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:03.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:02 smithi132 bash[18607]: cluster 2024-04-25T20:27:01.483682+0000 mgr.x (mgr.14254) 1034 : cluster [DBG] pgmap v842: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:03.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:02 smithi132 bash[18607]: cluster 2024-04-25T20:27:01.483682+0000 mgr.x (mgr.14254) 1034 : cluster [DBG] pgmap v842: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:03.864 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:27:04.870 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:04 smithi132 bash[18607]: cluster 2024-04-25T20:27:03.485551+0000 mgr.x (mgr.14254) 1035 : cluster [DBG] pgmap v843: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:04.871 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:04 smithi132 bash[18607]: cluster 2024-04-25T20:27:03.485551+0000 mgr.x (mgr.14254) 1035 : cluster [DBG] pgmap v843: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:06.731 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:27:06.731 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:27:06.744 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:06 smithi132 bash[18607]: cluster 2024-04-25T20:27:05.486966+0000 mgr.x (mgr.14254) 1036 : cluster [DBG] pgmap v844: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:06.744 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:06 smithi132 bash[18607]: cluster 2024-04-25T20:27:05.486966+0000 mgr.x (mgr.14254) 1036 : cluster [DBG] pgmap v844: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:07.465 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:27:07.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:07 smithi132 bash[18607]: audit 2024-04-25T20:27:06.728393+0000 mgr.x (mgr.14254) 1037 : audit [DBG] from='client.14786 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:27:07.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:07 smithi132 bash[18607]: audit 2024-04-25T20:27:06.728393+0000 mgr.x (mgr.14254) 1037 : audit [DBG] from='client.14786 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:27:08.466 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:27:08.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:08 smithi132 bash[18607]: cluster 2024-04-25T20:27:07.487915+0000 mgr.x (mgr.14254) 1038 : cluster [DBG] pgmap v845: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:08.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:08 smithi132 bash[18607]: cluster 2024-04-25T20:27:07.487915+0000 mgr.x (mgr.14254) 1038 : cluster [DBG] pgmap v845: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:11.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:10 smithi132 bash[18607]: cluster 2024-04-25T20:27:09.489745+0000 mgr.x (mgr.14254) 1039 : cluster [DBG] pgmap v846: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:11.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:10 smithi132 bash[18607]: cluster 2024-04-25T20:27:09.489745+0000 mgr.x (mgr.14254) 1039 : cluster [DBG] pgmap v846: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:13.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:12 smithi132 bash[18607]: cluster 2024-04-25T20:27:11.491200+0000 mgr.x (mgr.14254) 1040 : cluster [DBG] pgmap v847: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:13.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:12 smithi132 bash[18607]: cluster 2024-04-25T20:27:11.491200+0000 mgr.x (mgr.14254) 1040 : cluster [DBG] pgmap v847: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:13.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:12 smithi132 bash[18607]: audit 2024-04-25T20:27:11.739250+0000 mon.a (mon.0) 501 : audit [DBG] from='client.? 172.17.0.2:0/1689930434' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:27:13.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:12 smithi132 bash[18607]: audit 2024-04-25T20:27:11.739250+0000 mon.a (mon.0) 501 : audit [DBG] from='client.? 172.17.0.2:0/1689930434' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:27:13.232 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:27:14.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:14 smithi132 bash[18607]: cluster 2024-04-25T20:27:13.493116+0000 mgr.x (mgr.14254) 1041 : cluster [DBG] pgmap v848: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:14.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:14 smithi132 bash[18607]: cluster 2024-04-25T20:27:13.493116+0000 mgr.x (mgr.14254) 1041 : cluster [DBG] pgmap v848: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:16.051 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:27:16.051 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:27:16.701 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:27:17.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:16 smithi132 bash[18607]: cluster 2024-04-25T20:27:15.494580+0000 mgr.x (mgr.14254) 1042 : cluster [DBG] pgmap v849: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:17.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:16 smithi132 bash[18607]: cluster 2024-04-25T20:27:15.494580+0000 mgr.x (mgr.14254) 1042 : cluster [DBG] pgmap v849: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:17.702 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:27:18.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:17 smithi132 bash[18607]: audit 2024-04-25T20:27:16.048320+0000 mgr.x (mgr.14254) 1043 : audit [DBG] from='client.14790 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:27:18.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:17 smithi132 bash[18607]: audit 2024-04-25T20:27:16.048320+0000 mgr.x (mgr.14254) 1043 : audit [DBG] from='client.14790 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:27:19.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:18 smithi132 bash[18607]: cluster 2024-04-25T20:27:17.496007+0000 mgr.x (mgr.14254) 1044 : cluster [DBG] pgmap v850: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:19.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:18 smithi132 bash[18607]: cluster 2024-04-25T20:27:17.496007+0000 mgr.x (mgr.14254) 1044 : cluster [DBG] pgmap v850: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:21.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:20 smithi132 bash[18607]: cluster 2024-04-25T20:27:19.497853+0000 mgr.x (mgr.14254) 1045 : cluster [DBG] pgmap v851: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:21.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:20 smithi132 bash[18607]: cluster 2024-04-25T20:27:19.497853+0000 mgr.x (mgr.14254) 1045 : cluster [DBG] pgmap v851: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:22.465 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:27:22.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:22 smithi132 bash[18607]: cluster 2024-04-25T20:27:21.499372+0000 mgr.x (mgr.14254) 1046 : cluster [DBG] pgmap v852: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:22.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:22 smithi132 bash[18607]: cluster 2024-04-25T20:27:21.499372+0000 mgr.x (mgr.14254) 1046 : cluster [DBG] pgmap v852: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:24.929 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:24 smithi132 bash[18607]: cluster 2024-04-25T20:27:23.500435+0000 mgr.x (mgr.14254) 1047 : cluster [DBG] pgmap v853: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:24.930 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:24 smithi132 bash[18607]: cluster 2024-04-25T20:27:23.500435+0000 mgr.x (mgr.14254) 1047 : cluster [DBG] pgmap v853: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:25.328 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:27:25.328 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:27:25.988 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:27:26.989 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:27:27.000 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:26 smithi132 bash[18607]: audit 2024-04-25T20:27:25.328813+0000 mgr.x (mgr.14254) 1048 : audit [DBG] from='client.14792 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:27:27.000 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:26 smithi132 bash[18607]: audit 2024-04-25T20:27:25.328813+0000 mgr.x (mgr.14254) 1048 : audit [DBG] from='client.14792 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:27:27.000 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:26 smithi132 bash[18607]: cluster 2024-04-25T20:27:25.501804+0000 mgr.x (mgr.14254) 1049 : cluster [DBG] pgmap v854: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:27.000 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:26 smithi132 bash[18607]: cluster 2024-04-25T20:27:25.501804+0000 mgr.x (mgr.14254) 1049 : cluster [DBG] pgmap v854: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:29.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:28 smithi132 bash[18607]: cluster 2024-04-25T20:27:27.503239+0000 mgr.x (mgr.14254) 1050 : cluster [DBG] pgmap v855: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:29.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:28 smithi132 bash[18607]: cluster 2024-04-25T20:27:27.503239+0000 mgr.x (mgr.14254) 1050 : cluster [DBG] pgmap v855: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:31.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:30 smithi132 bash[18607]: cluster 2024-04-25T20:27:29.505140+0000 mgr.x (mgr.14254) 1051 : cluster [DBG] pgmap v856: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:31.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:30 smithi132 bash[18607]: cluster 2024-04-25T20:27:29.505140+0000 mgr.x (mgr.14254) 1051 : cluster [DBG] pgmap v856: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:31.746 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:27:32.944 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:32 smithi132 bash[18607]: cluster 2024-04-25T20:27:31.506505+0000 mgr.x (mgr.14254) 1052 : cluster [DBG] pgmap v857: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:32.945 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:32 smithi132 bash[18607]: cluster 2024-04-25T20:27:31.506505+0000 mgr.x (mgr.14254) 1052 : cluster [DBG] pgmap v857: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:34.551 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:27:34.551 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:27:35.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:34 smithi132 bash[18607]: cluster 2024-04-25T20:27:33.507986+0000 mgr.x (mgr.14254) 1053 : cluster [DBG] pgmap v858: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:35.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:34 smithi132 bash[18607]: cluster 2024-04-25T20:27:33.507986+0000 mgr.x (mgr.14254) 1053 : cluster [DBG] pgmap v858: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:35.299 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:27:36.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:35 smithi132 bash[18607]: audit 2024-04-25T20:27:34.550961+0000 mgr.x (mgr.14254) 1054 : audit [DBG] from='client.14794 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:27:36.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:35 smithi132 bash[18607]: audit 2024-04-25T20:27:34.550961+0000 mgr.x (mgr.14254) 1054 : audit [DBG] from='client.14794 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:27:36.300 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:27:37.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:36 smithi132 bash[18607]: cluster 2024-04-25T20:27:35.509501+0000 mgr.x (mgr.14254) 1055 : cluster [DBG] pgmap v859: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:37.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:36 smithi132 bash[18607]: cluster 2024-04-25T20:27:35.509501+0000 mgr.x (mgr.14254) 1055 : cluster [DBG] pgmap v859: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:39.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:38 smithi132 bash[18607]: cluster 2024-04-25T20:27:37.510578+0000 mgr.x (mgr.14254) 1056 : cluster [DBG] pgmap v860: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:39.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:38 smithi132 bash[18607]: cluster 2024-04-25T20:27:37.510578+0000 mgr.x (mgr.14254) 1056 : cluster [DBG] pgmap v860: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:41.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:40 smithi132 bash[18607]: cluster 2024-04-25T20:27:39.512431+0000 mgr.x (mgr.14254) 1057 : cluster [DBG] pgmap v861: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:41.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:40 smithi132 bash[18607]: cluster 2024-04-25T20:27:39.512431+0000 mgr.x (mgr.14254) 1057 : cluster [DBG] pgmap v861: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:41.055 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:27:43.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:42 smithi132 bash[18607]: cluster 2024-04-25T20:27:41.513895+0000 mgr.x (mgr.14254) 1058 : cluster [DBG] pgmap v862: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:43.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:42 smithi132 bash[18607]: cluster 2024-04-25T20:27:41.513895+0000 mgr.x (mgr.14254) 1058 : cluster [DBG] pgmap v862: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:43.844 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:27:43.844 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:27:44.534 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:27:45.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:44 smithi132 bash[18607]: cluster 2024-04-25T20:27:43.514826+0000 mgr.x (mgr.14254) 1059 : cluster [DBG] pgmap v863: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:45.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:44 smithi132 bash[18607]: cluster 2024-04-25T20:27:43.514826+0000 mgr.x (mgr.14254) 1059 : cluster [DBG] pgmap v863: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:45.536 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:27:46.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:45 smithi132 bash[18607]: audit 2024-04-25T20:27:43.844002+0000 mgr.x (mgr.14254) 1060 : audit [DBG] from='client.14796 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:27:46.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:45 smithi132 bash[18607]: audit 2024-04-25T20:27:43.844002+0000 mgr.x (mgr.14254) 1060 : audit [DBG] from='client.14796 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:27:47.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:46 smithi132 bash[18607]: cluster 2024-04-25T20:27:45.516226+0000 mgr.x (mgr.14254) 1061 : cluster [DBG] pgmap v864: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:47.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:46 smithi132 bash[18607]: cluster 2024-04-25T20:27:45.516226+0000 mgr.x (mgr.14254) 1061 : cluster [DBG] pgmap v864: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:49.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:48 smithi132 bash[18607]: cluster 2024-04-25T20:27:47.517625+0000 mgr.x (mgr.14254) 1062 : cluster [DBG] pgmap v865: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:49.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:48 smithi132 bash[18607]: cluster 2024-04-25T20:27:47.517625+0000 mgr.x (mgr.14254) 1062 : cluster [DBG] pgmap v865: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:50.293 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:27:51.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:50 smithi132 bash[18607]: cluster 2024-04-25T20:27:49.519476+0000 mgr.x (mgr.14254) 1063 : cluster [DBG] pgmap v866: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:51.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:50 smithi132 bash[18607]: cluster 2024-04-25T20:27:49.519476+0000 mgr.x (mgr.14254) 1063 : cluster [DBG] pgmap v866: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:53.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:52 smithi132 bash[18607]: cluster 2024-04-25T20:27:51.520947+0000 mgr.x (mgr.14254) 1064 : cluster [DBG] pgmap v867: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:53.026 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:52 smithi132 bash[18607]: cluster 2024-04-25T20:27:51.520947+0000 mgr.x (mgr.14254) 1064 : cluster [DBG] pgmap v867: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:53.043 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:27:53.044 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:27:53.636 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:27:54.638 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:27:55.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:54 smithi132 bash[18607]: audit 2024-04-25T20:27:53.043988+0000 mgr.x (mgr.14254) 1065 : audit [DBG] from='client.14798 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:27:55.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:54 smithi132 bash[18607]: audit 2024-04-25T20:27:53.043988+0000 mgr.x (mgr.14254) 1065 : audit [DBG] from='client.14798 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:27:55.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:54 smithi132 bash[18607]: cluster 2024-04-25T20:27:53.522789+0000 mgr.x (mgr.14254) 1066 : cluster [DBG] pgmap v868: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:55.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:54 smithi132 bash[18607]: cluster 2024-04-25T20:27:53.522789+0000 mgr.x (mgr.14254) 1066 : cluster [DBG] pgmap v868: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:57.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:56 smithi132 bash[18607]: cluster 2024-04-25T20:27:55.524213+0000 mgr.x (mgr.14254) 1067 : cluster [DBG] pgmap v869: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:57.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:56 smithi132 bash[18607]: cluster 2024-04-25T20:27:55.524213+0000 mgr.x (mgr.14254) 1067 : cluster [DBG] pgmap v869: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:59.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:58 smithi132 bash[18607]: cluster 2024-04-25T20:27:57.525630+0000 mgr.x (mgr.14254) 1068 : cluster [DBG] pgmap v870: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:59.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:27:58 smithi132 bash[18607]: cluster 2024-04-25T20:27:57.525630+0000 mgr.x (mgr.14254) 1068 : cluster [DBG] pgmap v870: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:27:59.401 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:28:00.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:00 smithi132 bash[18607]: cluster 2024-04-25T20:27:59.527516+0000 mgr.x (mgr.14254) 1069 : cluster [DBG] pgmap v871: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:00.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:00 smithi132 bash[18607]: cluster 2024-04-25T20:27:59.527516+0000 mgr.x (mgr.14254) 1069 : cluster [DBG] pgmap v871: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:02.187 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:28:02.187 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:28:02.924 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:02 smithi132 bash[18607]: cluster 2024-04-25T20:28:01.528422+0000 mgr.x (mgr.14254) 1070 : cluster [DBG] pgmap v872: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:02.924 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:02 smithi132 bash[18607]: cluster 2024-04-25T20:28:01.528422+0000 mgr.x (mgr.14254) 1070 : cluster [DBG] pgmap v872: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:02.925 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:28:03.926 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:28:03.937 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:03 smithi132 bash[18607]: audit 2024-04-25T20:28:02.184594+0000 mgr.x (mgr.14254) 1071 : audit [DBG] from='client.14800 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:28:03.937 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:03 smithi132 bash[18607]: audit 2024-04-25T20:28:02.184594+0000 mgr.x (mgr.14254) 1071 : audit [DBG] from='client.14800 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:28:05.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:04 smithi132 bash[18607]: cluster 2024-04-25T20:28:03.530330+0000 mgr.x (mgr.14254) 1072 : cluster [DBG] pgmap v873: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:05.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:04 smithi132 bash[18607]: cluster 2024-04-25T20:28:03.530330+0000 mgr.x (mgr.14254) 1072 : cluster [DBG] pgmap v873: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:07.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:06 smithi132 bash[18607]: cluster 2024-04-25T20:28:05.531717+0000 mgr.x (mgr.14254) 1073 : cluster [DBG] pgmap v874: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:07.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:06 smithi132 bash[18607]: cluster 2024-04-25T20:28:05.531717+0000 mgr.x (mgr.14254) 1073 : cluster [DBG] pgmap v874: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:08.678 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:28:09.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:08 smithi132 bash[18607]: cluster 2024-04-25T20:28:07.533108+0000 mgr.x (mgr.14254) 1074 : cluster [DBG] pgmap v875: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:09.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:08 smithi132 bash[18607]: cluster 2024-04-25T20:28:07.533108+0000 mgr.x (mgr.14254) 1074 : cluster [DBG] pgmap v875: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:10.948 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:10 smithi132 bash[18607]: cluster 2024-04-25T20:28:09.534285+0000 mgr.x (mgr.14254) 1075 : cluster [DBG] pgmap v876: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:10.948 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:10 smithi132 bash[18607]: cluster 2024-04-25T20:28:09.534285+0000 mgr.x (mgr.14254) 1075 : cluster [DBG] pgmap v876: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:11.359 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:28:11.359 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:28:12.113 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:28:13.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:12 smithi132 bash[18607]: audit 2024-04-25T20:28:11.356753+0000 mgr.x (mgr.14254) 1076 : audit [DBG] from='client.14802 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:28:13.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:12 smithi132 bash[18607]: audit 2024-04-25T20:28:11.356753+0000 mgr.x (mgr.14254) 1076 : audit [DBG] from='client.14802 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:28:13.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:12 smithi132 bash[18607]: cluster 2024-04-25T20:28:11.535604+0000 mgr.x (mgr.14254) 1077 : cluster [DBG] pgmap v877: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:13.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:12 smithi132 bash[18607]: cluster 2024-04-25T20:28:11.535604+0000 mgr.x (mgr.14254) 1077 : cluster [DBG] pgmap v877: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:13.114 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:28:15.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:14 smithi132 bash[18607]: cluster 2024-04-25T20:28:13.537424+0000 mgr.x (mgr.14254) 1078 : cluster [DBG] pgmap v878: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:15.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:14 smithi132 bash[18607]: cluster 2024-04-25T20:28:13.537424+0000 mgr.x (mgr.14254) 1078 : cluster [DBG] pgmap v878: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:17.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:16 smithi132 bash[18607]: cluster 2024-04-25T20:28:15.538859+0000 mgr.x (mgr.14254) 1079 : cluster [DBG] pgmap v879: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:17.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:16 smithi132 bash[18607]: cluster 2024-04-25T20:28:15.538859+0000 mgr.x (mgr.14254) 1079 : cluster [DBG] pgmap v879: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:17.871 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:28:19.012 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:18 smithi132 bash[18607]: cluster 2024-04-25T20:28:17.540318+0000 mgr.x (mgr.14254) 1080 : cluster [DBG] pgmap v880: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:19.012 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:18 smithi132 bash[18607]: cluster 2024-04-25T20:28:17.540318+0000 mgr.x (mgr.14254) 1080 : cluster [DBG] pgmap v880: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:20.651 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:28:20.651 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:28:21.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:20 smithi132 bash[18607]: cluster 2024-04-25T20:28:19.541523+0000 mgr.x (mgr.14254) 1081 : cluster [DBG] pgmap v881: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:21.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:20 smithi132 bash[18607]: cluster 2024-04-25T20:28:19.541523+0000 mgr.x (mgr.14254) 1081 : cluster [DBG] pgmap v881: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:21.364 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:28:22.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:21 smithi132 bash[18607]: audit 2024-04-25T20:28:20.651492+0000 mgr.x (mgr.14254) 1082 : audit [DBG] from='client.14804 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:28:22.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:21 smithi132 bash[18607]: audit 2024-04-25T20:28:20.651492+0000 mgr.x (mgr.14254) 1082 : audit [DBG] from='client.14804 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:28:22.365 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:28:23.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:22 smithi132 bash[18607]: cluster 2024-04-25T20:28:21.543058+0000 mgr.x (mgr.14254) 1083 : cluster [DBG] pgmap v882: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:23.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:22 smithi132 bash[18607]: cluster 2024-04-25T20:28:21.543058+0000 mgr.x (mgr.14254) 1083 : cluster [DBG] pgmap v882: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:23.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:22 smithi132 bash[18607]: audit 2024-04-25T20:28:22.127275+0000 mon.a (mon.0) 502 : audit [DBG] from='client.? 172.17.0.2:0/1267283208' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:28:23.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:22 smithi132 bash[18607]: audit 2024-04-25T20:28:22.127275+0000 mon.a (mon.0) 502 : audit [DBG] from='client.? 172.17.0.2:0/1267283208' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:28:25.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:24 smithi132 bash[18607]: cluster 2024-04-25T20:28:23.544914+0000 mgr.x (mgr.14254) 1084 : cluster [DBG] pgmap v883: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:25.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:24 smithi132 bash[18607]: cluster 2024-04-25T20:28:23.544914+0000 mgr.x (mgr.14254) 1084 : cluster [DBG] pgmap v883: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:27.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:26 smithi132 bash[18607]: cluster 2024-04-25T20:28:25.546287+0000 mgr.x (mgr.14254) 1085 : cluster [DBG] pgmap v884: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:27.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:26 smithi132 bash[18607]: cluster 2024-04-25T20:28:25.546287+0000 mgr.x (mgr.14254) 1085 : cluster [DBG] pgmap v884: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:27.120 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:28:29.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:28 smithi132 bash[18607]: cluster 2024-04-25T20:28:27.547711+0000 mgr.x (mgr.14254) 1086 : cluster [DBG] pgmap v885: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:29.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:28 smithi132 bash[18607]: cluster 2024-04-25T20:28:27.547711+0000 mgr.x (mgr.14254) 1086 : cluster [DBG] pgmap v885: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:29.924 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:28:29.924 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:28:30.602 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:28:31.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:30 smithi132 bash[18607]: cluster 2024-04-25T20:28:29.548722+0000 mgr.x (mgr.14254) 1087 : cluster [DBG] pgmap v886: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:31.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:30 smithi132 bash[18607]: cluster 2024-04-25T20:28:29.548722+0000 mgr.x (mgr.14254) 1087 : cluster [DBG] pgmap v886: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:31.603 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:28:32.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:31 smithi132 bash[18607]: audit 2024-04-25T20:28:29.921437+0000 mgr.x (mgr.14254) 1088 : audit [DBG] from='client.14808 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:28:32.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:31 smithi132 bash[18607]: audit 2024-04-25T20:28:29.921437+0000 mgr.x (mgr.14254) 1088 : audit [DBG] from='client.14808 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:28:33.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:32 smithi132 bash[18607]: cluster 2024-04-25T20:28:31.550193+0000 mgr.x (mgr.14254) 1089 : cluster [DBG] pgmap v887: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:33.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:32 smithi132 bash[18607]: cluster 2024-04-25T20:28:31.550193+0000 mgr.x (mgr.14254) 1089 : cluster [DBG] pgmap v887: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:35.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:34 smithi132 bash[18607]: cluster 2024-04-25T20:28:33.552089+0000 mgr.x (mgr.14254) 1090 : cluster [DBG] pgmap v888: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:35.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:34 smithi132 bash[18607]: cluster 2024-04-25T20:28:33.552089+0000 mgr.x (mgr.14254) 1090 : cluster [DBG] pgmap v888: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:36.365 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:28:37.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:36 smithi132 bash[18607]: cluster 2024-04-25T20:28:35.553519+0000 mgr.x (mgr.14254) 1091 : cluster [DBG] pgmap v889: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:37.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:36 smithi132 bash[18607]: cluster 2024-04-25T20:28:35.553519+0000 mgr.x (mgr.14254) 1091 : cluster [DBG] pgmap v889: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:39.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:38 smithi132 bash[18607]: cluster 2024-04-25T20:28:37.554301+0000 mgr.x (mgr.14254) 1092 : cluster [DBG] pgmap v890: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:39.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:38 smithi132 bash[18607]: cluster 2024-04-25T20:28:37.554301+0000 mgr.x (mgr.14254) 1092 : cluster [DBG] pgmap v890: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:39.283 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:28:39.283 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:28:40.031 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:28:41.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:40 smithi132 bash[18607]: audit 2024-04-25T20:28:39.280612+0000 mgr.x (mgr.14254) 1093 : audit [DBG] from='client.14810 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:28:41.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:40 smithi132 bash[18607]: audit 2024-04-25T20:28:39.280612+0000 mgr.x (mgr.14254) 1093 : audit [DBG] from='client.14810 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:28:41.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:40 smithi132 bash[18607]: cluster 2024-04-25T20:28:39.556169+0000 mgr.x (mgr.14254) 1094 : cluster [DBG] pgmap v891: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:41.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:40 smithi132 bash[18607]: cluster 2024-04-25T20:28:39.556169+0000 mgr.x (mgr.14254) 1094 : cluster [DBG] pgmap v891: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:41.032 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:28:43.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:42 smithi132 bash[18607]: cluster 2024-04-25T20:28:41.557624+0000 mgr.x (mgr.14254) 1095 : cluster [DBG] pgmap v892: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:43.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:42 smithi132 bash[18607]: cluster 2024-04-25T20:28:41.557624+0000 mgr.x (mgr.14254) 1095 : cluster [DBG] pgmap v892: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:45.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:44 smithi132 bash[18607]: cluster 2024-04-25T20:28:43.559471+0000 mgr.x (mgr.14254) 1096 : cluster [DBG] pgmap v893: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:45.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:44 smithi132 bash[18607]: cluster 2024-04-25T20:28:43.559471+0000 mgr.x (mgr.14254) 1096 : cluster [DBG] pgmap v893: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:45.793 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:28:46.950 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:46 smithi132 bash[18607]: cluster 2024-04-25T20:28:45.560962+0000 mgr.x (mgr.14254) 1097 : cluster [DBG] pgmap v894: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:46.951 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:46 smithi132 bash[18607]: cluster 2024-04-25T20:28:45.560962+0000 mgr.x (mgr.14254) 1097 : cluster [DBG] pgmap v894: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:48.569 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:28:48.569 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:28:49.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:48 smithi132 bash[18607]: cluster 2024-04-25T20:28:47.561824+0000 mgr.x (mgr.14254) 1098 : cluster [DBG] pgmap v895: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:49.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:48 smithi132 bash[18607]: cluster 2024-04-25T20:28:47.561824+0000 mgr.x (mgr.14254) 1098 : cluster [DBG] pgmap v895: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:49.242 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:28:50.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:49 smithi132 bash[18607]: audit 2024-04-25T20:28:48.568615+0000 mgr.x (mgr.14254) 1099 : audit [DBG] from='client.14812 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:28:50.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:49 smithi132 bash[18607]: audit 2024-04-25T20:28:48.568615+0000 mgr.x (mgr.14254) 1099 : audit [DBG] from='client.14812 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:28:50.244 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:28:51.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:50 smithi132 bash[18607]: cluster 2024-04-25T20:28:49.563700+0000 mgr.x (mgr.14254) 1100 : cluster [DBG] pgmap v896: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:51.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:50 smithi132 bash[18607]: cluster 2024-04-25T20:28:49.563700+0000 mgr.x (mgr.14254) 1100 : cluster [DBG] pgmap v896: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:53.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:52 smithi132 bash[18607]: cluster 2024-04-25T20:28:51.565231+0000 mgr.x (mgr.14254) 1101 : cluster [DBG] pgmap v897: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:53.023 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:52 smithi132 bash[18607]: cluster 2024-04-25T20:28:51.565231+0000 mgr.x (mgr.14254) 1101 : cluster [DBG] pgmap v897: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:55.001 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:28:55.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:54 smithi132 bash[18607]: cluster 2024-04-25T20:28:53.567209+0000 mgr.x (mgr.14254) 1102 : cluster [DBG] pgmap v898: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:55.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:54 smithi132 bash[18607]: cluster 2024-04-25T20:28:53.567209+0000 mgr.x (mgr.14254) 1102 : cluster [DBG] pgmap v898: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:57.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:56 smithi132 bash[18607]: cluster 2024-04-25T20:28:55.568801+0000 mgr.x (mgr.14254) 1103 : cluster [DBG] pgmap v899: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:57.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:56 smithi132 bash[18607]: cluster 2024-04-25T20:28:55.568801+0000 mgr.x (mgr.14254) 1103 : cluster [DBG] pgmap v899: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:57.833 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:28:57.833 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:28:58.497 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:28:58.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:58 smithi132 bash[18607]: cluster 2024-04-25T20:28:57.570310+0000 mgr.x (mgr.14254) 1104 : cluster [DBG] pgmap v900: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:58.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:58 smithi132 bash[18607]: cluster 2024-04-25T20:28:57.570310+0000 mgr.x (mgr.14254) 1104 : cluster [DBG] pgmap v900: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:28:59.498 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:28:59.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:59 smithi132 bash[18607]: audit 2024-04-25T20:28:57.834094+0000 mgr.x (mgr.14254) 1105 : audit [DBG] from='client.14814 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:28:59.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:28:59 smithi132 bash[18607]: audit 2024-04-25T20:28:57.834094+0000 mgr.x (mgr.14254) 1105 : audit [DBG] from='client.14814 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:29:01.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:00 smithi132 bash[18607]: cluster 2024-04-25T20:28:59.571583+0000 mgr.x (mgr.14254) 1106 : cluster [DBG] pgmap v901: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:01.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:00 smithi132 bash[18607]: cluster 2024-04-25T20:28:59.571583+0000 mgr.x (mgr.14254) 1106 : cluster [DBG] pgmap v901: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:03.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:02 smithi132 bash[18607]: cluster 2024-04-25T20:29:01.572977+0000 mgr.x (mgr.14254) 1107 : cluster [DBG] pgmap v902: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:03.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:02 smithi132 bash[18607]: cluster 2024-04-25T20:29:01.572977+0000 mgr.x (mgr.14254) 1107 : cluster [DBG] pgmap v902: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:04.257 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:29:05.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:04 smithi132 bash[18607]: cluster 2024-04-25T20:29:03.574932+0000 mgr.x (mgr.14254) 1108 : cluster [DBG] pgmap v903: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:05.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:04 smithi132 bash[18607]: cluster 2024-04-25T20:29:03.574932+0000 mgr.x (mgr.14254) 1108 : cluster [DBG] pgmap v903: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:07.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:06 smithi132 bash[18607]: cluster 2024-04-25T20:29:05.576472+0000 mgr.x (mgr.14254) 1109 : cluster [DBG] pgmap v904: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:07.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:06 smithi132 bash[18607]: cluster 2024-04-25T20:29:05.576472+0000 mgr.x (mgr.14254) 1109 : cluster [DBG] pgmap v904: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:07.054 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:29:07.054 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:29:07.775 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:29:08.776 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:29:09.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:08 smithi132 bash[18607]: audit 2024-04-25T20:29:07.054712+0000 mgr.x (mgr.14254) 1110 : audit [DBG] from='client.14816 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:29:09.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:08 smithi132 bash[18607]: audit 2024-04-25T20:29:07.054712+0000 mgr.x (mgr.14254) 1110 : audit [DBG] from='client.14816 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:29:09.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:08 smithi132 bash[18607]: cluster 2024-04-25T20:29:07.577921+0000 mgr.x (mgr.14254) 1111 : cluster [DBG] pgmap v905: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:09.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:08 smithi132 bash[18607]: cluster 2024-04-25T20:29:07.577921+0000 mgr.x (mgr.14254) 1111 : cluster [DBG] pgmap v905: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:11.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:10 smithi132 bash[18607]: cluster 2024-04-25T20:29:09.579782+0000 mgr.x (mgr.14254) 1112 : cluster [DBG] pgmap v906: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:11.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:10 smithi132 bash[18607]: cluster 2024-04-25T20:29:09.579782+0000 mgr.x (mgr.14254) 1112 : cluster [DBG] pgmap v906: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:12.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:11 smithi132 bash[18607]: cluster 2024-04-25T20:29:11.581363+0000 mgr.x (mgr.14254) 1113 : cluster [DBG] pgmap v907: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:12.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:11 smithi132 bash[18607]: cluster 2024-04-25T20:29:11.581363+0000 mgr.x (mgr.14254) 1113 : cluster [DBG] pgmap v907: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:13.528 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:29:15.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:14 smithi132 bash[18607]: cluster 2024-04-25T20:29:13.582568+0000 mgr.x (mgr.14254) 1114 : cluster [DBG] pgmap v908: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:15.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:14 smithi132 bash[18607]: cluster 2024-04-25T20:29:13.582568+0000 mgr.x (mgr.14254) 1114 : cluster [DBG] pgmap v908: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:16.233 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:29:16.233 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:29:16.913 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:16 smithi132 bash[18607]: cluster 2024-04-25T20:29:15.583966+0000 mgr.x (mgr.14254) 1115 : cluster [DBG] pgmap v909: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:16.913 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:16 smithi132 bash[18607]: cluster 2024-04-25T20:29:15.583966+0000 mgr.x (mgr.14254) 1115 : cluster [DBG] pgmap v909: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:16.914 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:29:17.915 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:29:17.927 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:17 smithi132 bash[18607]: audit 2024-04-25T20:29:16.232266+0000 mgr.x (mgr.14254) 1116 : audit [DBG] from='client.14818 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:29:17.927 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:17 smithi132 bash[18607]: audit 2024-04-25T20:29:16.232266+0000 mgr.x (mgr.14254) 1116 : audit [DBG] from='client.14818 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:29:19.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:18 smithi132 bash[18607]: cluster 2024-04-25T20:29:17.585437+0000 mgr.x (mgr.14254) 1117 : cluster [DBG] pgmap v910: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:19.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:18 smithi132 bash[18607]: cluster 2024-04-25T20:29:17.585437+0000 mgr.x (mgr.14254) 1117 : cluster [DBG] pgmap v910: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:21.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:20 smithi132 bash[18607]: cluster 2024-04-25T20:29:19.587430+0000 mgr.x (mgr.14254) 1118 : cluster [DBG] pgmap v911: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:21.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:20 smithi132 bash[18607]: cluster 2024-04-25T20:29:19.587430+0000 mgr.x (mgr.14254) 1118 : cluster [DBG] pgmap v911: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:22.664 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:29:23.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:22 smithi132 bash[18607]: cluster 2024-04-25T20:29:21.588902+0000 mgr.x (mgr.14254) 1119 : cluster [DBG] pgmap v912: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:23.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:22 smithi132 bash[18607]: cluster 2024-04-25T20:29:21.588902+0000 mgr.x (mgr.14254) 1119 : cluster [DBG] pgmap v912: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:24.927 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:24 smithi132 bash[18607]: cluster 2024-04-25T20:29:23.589847+0000 mgr.x (mgr.14254) 1120 : cluster [DBG] pgmap v913: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:24.927 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:24 smithi132 bash[18607]: cluster 2024-04-25T20:29:23.589847+0000 mgr.x (mgr.14254) 1120 : cluster [DBG] pgmap v913: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:25.341 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:29:25.342 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:29:26.007 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:29:27.009 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:29:27.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:26 smithi132 bash[18607]: audit 2024-04-25T20:29:25.339131+0000 mgr.x (mgr.14254) 1121 : audit [DBG] from='client.14820 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:29:27.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:26 smithi132 bash[18607]: audit 2024-04-25T20:29:25.339131+0000 mgr.x (mgr.14254) 1121 : audit [DBG] from='client.14820 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:29:27.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:26 smithi132 bash[18607]: cluster 2024-04-25T20:29:25.591407+0000 mgr.x (mgr.14254) 1122 : cluster [DBG] pgmap v914: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:27.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:26 smithi132 bash[18607]: cluster 2024-04-25T20:29:25.591407+0000 mgr.x (mgr.14254) 1122 : cluster [DBG] pgmap v914: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:29.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:28 smithi132 bash[18607]: cluster 2024-04-25T20:29:27.592804+0000 mgr.x (mgr.14254) 1123 : cluster [DBG] pgmap v915: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:29.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:28 smithi132 bash[18607]: cluster 2024-04-25T20:29:27.592804+0000 mgr.x (mgr.14254) 1123 : cluster [DBG] pgmap v915: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:31.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:30 smithi132 bash[18607]: cluster 2024-04-25T20:29:29.594660+0000 mgr.x (mgr.14254) 1124 : cluster [DBG] pgmap v916: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:31.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:30 smithi132 bash[18607]: cluster 2024-04-25T20:29:29.594660+0000 mgr.x (mgr.14254) 1124 : cluster [DBG] pgmap v916: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:31.765 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:29:32.809 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:32 smithi132 bash[18607]: cluster 2024-04-25T20:29:31.596064+0000 mgr.x (mgr.14254) 1125 : cluster [DBG] pgmap v917: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:32.809 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:32 smithi132 bash[18607]: cluster 2024-04-25T20:29:31.596064+0000 mgr.x (mgr.14254) 1125 : cluster [DBG] pgmap v917: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:32.809 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:32 smithi132 bash[18607]: audit 2024-04-25T20:29:32.519879+0000 mon.a (mon.0) 503 : audit [DBG] from='client.? 172.17.0.2:0/3011304204' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:29:32.809 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:32 smithi132 bash[18607]: audit 2024-04-25T20:29:32.519879+0000 mon.a (mon.0) 503 : audit [DBG] from='client.? 172.17.0.2:0/3011304204' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:29:34.521 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:29:34.521 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:29:35.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:34 smithi132 bash[18607]: cluster 2024-04-25T20:29:33.597960+0000 mgr.x (mgr.14254) 1126 : cluster [DBG] pgmap v918: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:35.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:34 smithi132 bash[18607]: cluster 2024-04-25T20:29:33.597960+0000 mgr.x (mgr.14254) 1126 : cluster [DBG] pgmap v918: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:35.221 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:29:36.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:35 smithi132 bash[18607]: audit 2024-04-25T20:29:34.520535+0000 mgr.x (mgr.14254) 1127 : audit [DBG] from='client.14824 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:29:36.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:35 smithi132 bash[18607]: audit 2024-04-25T20:29:34.520535+0000 mgr.x (mgr.14254) 1127 : audit [DBG] from='client.14824 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:29:36.222 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:29:37.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:36 smithi132 bash[18607]: cluster 2024-04-25T20:29:35.599399+0000 mgr.x (mgr.14254) 1128 : cluster [DBG] pgmap v919: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:37.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:36 smithi132 bash[18607]: cluster 2024-04-25T20:29:35.599399+0000 mgr.x (mgr.14254) 1128 : cluster [DBG] pgmap v919: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:39.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:38 smithi132 bash[18607]: cluster 2024-04-25T20:29:37.600848+0000 mgr.x (mgr.14254) 1129 : cluster [DBG] pgmap v920: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:39.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:38 smithi132 bash[18607]: cluster 2024-04-25T20:29:37.600848+0000 mgr.x (mgr.14254) 1129 : cluster [DBG] pgmap v920: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:40.979 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:29:41.001 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:40 smithi132 bash[18607]: cluster 2024-04-25T20:29:39.602813+0000 mgr.x (mgr.14254) 1130 : cluster [DBG] pgmap v921: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:41.001 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:40 smithi132 bash[18607]: cluster 2024-04-25T20:29:39.602813+0000 mgr.x (mgr.14254) 1130 : cluster [DBG] pgmap v921: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:42.825 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:42 smithi132 bash[18607]: cluster 2024-04-25T20:29:41.604224+0000 mgr.x (mgr.14254) 1131 : cluster [DBG] pgmap v922: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:42.825 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:42 smithi132 bash[18607]: cluster 2024-04-25T20:29:41.604224+0000 mgr.x (mgr.14254) 1131 : cluster [DBG] pgmap v922: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:44.064 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:29:44.065 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:29:44.716 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:29:45.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:44 smithi132 bash[18607]: cluster 2024-04-25T20:29:43.606091+0000 mgr.x (mgr.14254) 1132 : cluster [DBG] pgmap v923: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:45.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:44 smithi132 bash[18607]: cluster 2024-04-25T20:29:43.606091+0000 mgr.x (mgr.14254) 1132 : cluster [DBG] pgmap v923: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:45.717 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:29:46.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:45 smithi132 bash[18607]: audit 2024-04-25T20:29:44.064313+0000 mgr.x (mgr.14254) 1133 : audit [DBG] from='client.14826 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:29:46.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:45 smithi132 bash[18607]: audit 2024-04-25T20:29:44.064313+0000 mgr.x (mgr.14254) 1133 : audit [DBG] from='client.14826 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:29:47.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:46 smithi132 bash[18607]: cluster 2024-04-25T20:29:45.607570+0000 mgr.x (mgr.14254) 1134 : cluster [DBG] pgmap v924: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:47.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:46 smithi132 bash[18607]: cluster 2024-04-25T20:29:45.607570+0000 mgr.x (mgr.14254) 1134 : cluster [DBG] pgmap v924: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:49.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:48 smithi132 bash[18607]: cluster 2024-04-25T20:29:47.608971+0000 mgr.x (mgr.14254) 1135 : cluster [DBG] pgmap v925: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:49.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:48 smithi132 bash[18607]: cluster 2024-04-25T20:29:47.608971+0000 mgr.x (mgr.14254) 1135 : cluster [DBG] pgmap v925: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:50.478 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:29:50.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:50 smithi132 bash[18607]: cluster 2024-04-25T20:29:49.610801+0000 mgr.x (mgr.14254) 1136 : cluster [DBG] pgmap v926: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:50.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:50 smithi132 bash[18607]: cluster 2024-04-25T20:29:49.610801+0000 mgr.x (mgr.14254) 1136 : cluster [DBG] pgmap v926: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:52.939 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:52 smithi132 bash[18607]: cluster 2024-04-25T20:29:51.611672+0000 mgr.x (mgr.14254) 1137 : cluster [DBG] pgmap v927: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:52.939 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:52 smithi132 bash[18607]: cluster 2024-04-25T20:29:51.611672+0000 mgr.x (mgr.14254) 1137 : cluster [DBG] pgmap v927: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:53.339 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:29:53.339 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:29:54.089 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:29:55.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:54 smithi132 bash[18607]: audit 2024-04-25T20:29:53.336982+0000 mgr.x (mgr.14254) 1138 : audit [DBG] from='client.14828 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:29:55.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:54 smithi132 bash[18607]: audit 2024-04-25T20:29:53.336982+0000 mgr.x (mgr.14254) 1138 : audit [DBG] from='client.14828 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:29:55.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:54 smithi132 bash[18607]: cluster 2024-04-25T20:29:53.613544+0000 mgr.x (mgr.14254) 1139 : cluster [DBG] pgmap v928: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:55.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:54 smithi132 bash[18607]: cluster 2024-04-25T20:29:53.613544+0000 mgr.x (mgr.14254) 1139 : cluster [DBG] pgmap v928: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:55.091 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:29:57.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:56 smithi132 bash[18607]: cluster 2024-04-25T20:29:55.614951+0000 mgr.x (mgr.14254) 1140 : cluster [DBG] pgmap v929: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:57.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:56 smithi132 bash[18607]: cluster 2024-04-25T20:29:55.614951+0000 mgr.x (mgr.14254) 1140 : cluster [DBG] pgmap v929: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:59.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:58 smithi132 bash[18607]: cluster 2024-04-25T20:29:57.616388+0000 mgr.x (mgr.14254) 1141 : cluster [DBG] pgmap v930: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:59.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:29:58 smithi132 bash[18607]: cluster 2024-04-25T20:29:57.616388+0000 mgr.x (mgr.14254) 1141 : cluster [DBG] pgmap v930: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:29:59.848 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:30:00.888 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:00 smithi132 bash[18607]: cluster 2024-04-25T20:29:59.618412+0000 mgr.x (mgr.14254) 1142 : cluster [DBG] pgmap v931: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:00.889 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:00 smithi132 bash[18607]: cluster 2024-04-25T20:29:59.618412+0000 mgr.x (mgr.14254) 1142 : cluster [DBG] pgmap v931: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:02.487 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:30:02.488 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:30:02.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:02 smithi132 bash[18607]: cluster 2024-04-25T20:30:01.620026+0000 mgr.x (mgr.14254) 1143 : cluster [DBG] pgmap v932: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:02.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:02 smithi132 bash[18607]: cluster 2024-04-25T20:30:01.620026+0000 mgr.x (mgr.14254) 1143 : cluster [DBG] pgmap v932: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:03.137 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:30:04.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:03 smithi132 bash[18607]: audit 2024-04-25T20:30:02.485377+0000 mgr.x (mgr.14254) 1144 : audit [DBG] from='client.14830 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:30:04.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:03 smithi132 bash[18607]: audit 2024-04-25T20:30:02.485377+0000 mgr.x (mgr.14254) 1144 : audit [DBG] from='client.14830 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:30:04.137 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:30:05.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:04 smithi132 bash[18607]: cluster 2024-04-25T20:30:03.621971+0000 mgr.x (mgr.14254) 1145 : cluster [DBG] pgmap v933: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:05.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:04 smithi132 bash[18607]: cluster 2024-04-25T20:30:03.621971+0000 mgr.x (mgr.14254) 1145 : cluster [DBG] pgmap v933: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:07.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:06 smithi132 bash[18607]: cluster 2024-04-25T20:30:05.623438+0000 mgr.x (mgr.14254) 1146 : cluster [DBG] pgmap v934: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:07.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:06 smithi132 bash[18607]: cluster 2024-04-25T20:30:05.623438+0000 mgr.x (mgr.14254) 1146 : cluster [DBG] pgmap v934: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:08.901 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:30:09.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:08 smithi132 bash[18607]: cluster 2024-04-25T20:30:07.624905+0000 mgr.x (mgr.14254) 1147 : cluster [DBG] pgmap v935: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:09.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:08 smithi132 bash[18607]: cluster 2024-04-25T20:30:07.624905+0000 mgr.x (mgr.14254) 1147 : cluster [DBG] pgmap v935: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:10.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:10 smithi132 bash[18607]: cluster 2024-04-25T20:30:09.626977+0000 mgr.x (mgr.14254) 1148 : cluster [DBG] pgmap v936: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:10.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:10 smithi132 bash[18607]: cluster 2024-04-25T20:30:09.626977+0000 mgr.x (mgr.14254) 1148 : cluster [DBG] pgmap v936: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:11.599 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:30:11.599 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:30:12.233 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:30:13.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:12 smithi132 bash[18607]: audit 2024-04-25T20:30:11.597034+0000 mgr.x (mgr.14254) 1149 : audit [DBG] from='client.14832 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:30:13.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:12 smithi132 bash[18607]: audit 2024-04-25T20:30:11.597034+0000 mgr.x (mgr.14254) 1149 : audit [DBG] from='client.14832 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:30:13.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:12 smithi132 bash[18607]: cluster 2024-04-25T20:30:11.627786+0000 mgr.x (mgr.14254) 1150 : cluster [DBG] pgmap v937: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:13.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:12 smithi132 bash[18607]: cluster 2024-04-25T20:30:11.627786+0000 mgr.x (mgr.14254) 1150 : cluster [DBG] pgmap v937: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:13.234 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:30:15.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:14 smithi132 bash[18607]: cluster 2024-04-25T20:30:13.629652+0000 mgr.x (mgr.14254) 1151 : cluster [DBG] pgmap v938: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:15.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:14 smithi132 bash[18607]: cluster 2024-04-25T20:30:13.629652+0000 mgr.x (mgr.14254) 1151 : cluster [DBG] pgmap v938: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:17.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:16 smithi132 bash[18607]: cluster 2024-04-25T20:30:15.631125+0000 mgr.x (mgr.14254) 1152 : cluster [DBG] pgmap v939: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:17.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:16 smithi132 bash[18607]: cluster 2024-04-25T20:30:15.631125+0000 mgr.x (mgr.14254) 1152 : cluster [DBG] pgmap v939: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:17.993 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:30:18.994 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:18 smithi132 bash[18607]: cluster 2024-04-25T20:30:17.632558+0000 mgr.x (mgr.14254) 1153 : cluster [DBG] pgmap v940: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:18.994 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:18 smithi132 bash[18607]: cluster 2024-04-25T20:30:17.632558+0000 mgr.x (mgr.14254) 1153 : cluster [DBG] pgmap v940: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:20.762 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:30:20.762 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:30:21.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:20 smithi132 bash[18607]: cluster 2024-04-25T20:30:19.633623+0000 mgr.x (mgr.14254) 1154 : cluster [DBG] pgmap v941: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:21.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:20 smithi132 bash[18607]: cluster 2024-04-25T20:30:19.633623+0000 mgr.x (mgr.14254) 1154 : cluster [DBG] pgmap v941: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:21.411 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:30:22.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:21 smithi132 bash[18607]: audit 2024-04-25T20:30:20.759835+0000 mgr.x (mgr.14254) 1155 : audit [DBG] from='client.14834 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:30:22.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:21 smithi132 bash[18607]: audit 2024-04-25T20:30:20.759835+0000 mgr.x (mgr.14254) 1155 : audit [DBG] from='client.14834 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:30:22.413 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:30:23.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:22 smithi132 bash[18607]: cluster 2024-04-25T20:30:21.635044+0000 mgr.x (mgr.14254) 1156 : cluster [DBG] pgmap v942: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:23.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:22 smithi132 bash[18607]: cluster 2024-04-25T20:30:21.635044+0000 mgr.x (mgr.14254) 1156 : cluster [DBG] pgmap v942: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:25.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:24 smithi132 bash[18607]: cluster 2024-04-25T20:30:23.636340+0000 mgr.x (mgr.14254) 1157 : cluster [DBG] pgmap v943: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:25.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:24 smithi132 bash[18607]: cluster 2024-04-25T20:30:23.636340+0000 mgr.x (mgr.14254) 1157 : cluster [DBG] pgmap v943: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:27.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:26 smithi132 bash[18607]: cluster 2024-04-25T20:30:25.637750+0000 mgr.x (mgr.14254) 1158 : cluster [DBG] pgmap v944: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:27.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:26 smithi132 bash[18607]: cluster 2024-04-25T20:30:25.637750+0000 mgr.x (mgr.14254) 1158 : cluster [DBG] pgmap v944: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:27.174 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:30:29.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:28 smithi132 bash[18607]: cluster 2024-04-25T20:30:27.639191+0000 mgr.x (mgr.14254) 1159 : cluster [DBG] pgmap v945: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:29.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:28 smithi132 bash[18607]: cluster 2024-04-25T20:30:27.639191+0000 mgr.x (mgr.14254) 1159 : cluster [DBG] pgmap v945: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:29.881 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:30:29.881 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:30:30.721 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:30:31.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:30 smithi132 bash[18607]: cluster 2024-04-25T20:30:29.640074+0000 mgr.x (mgr.14254) 1160 : cluster [DBG] pgmap v946: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:31.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:30 smithi132 bash[18607]: cluster 2024-04-25T20:30:29.640074+0000 mgr.x (mgr.14254) 1160 : cluster [DBG] pgmap v946: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:31.722 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:30:32.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:31 smithi132 bash[18607]: audit 2024-04-25T20:30:29.878877+0000 mgr.x (mgr.14254) 1161 : audit [DBG] from='client.14836 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:30:32.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:31 smithi132 bash[18607]: audit 2024-04-25T20:30:29.878877+0000 mgr.x (mgr.14254) 1161 : audit [DBG] from='client.14836 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:30:33.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:32 smithi132 bash[18607]: cluster 2024-04-25T20:30:31.641510+0000 mgr.x (mgr.14254) 1162 : cluster [DBG] pgmap v947: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:33.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:32 smithi132 bash[18607]: cluster 2024-04-25T20:30:31.641510+0000 mgr.x (mgr.14254) 1162 : cluster [DBG] pgmap v947: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:35.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:34 smithi132 bash[18607]: cluster 2024-04-25T20:30:33.643352+0000 mgr.x (mgr.14254) 1163 : cluster [DBG] pgmap v948: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:35.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:34 smithi132 bash[18607]: cluster 2024-04-25T20:30:33.643352+0000 mgr.x (mgr.14254) 1163 : cluster [DBG] pgmap v948: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:36.482 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:30:36.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:36 smithi132 bash[18607]: cluster 2024-04-25T20:30:35.644805+0000 mgr.x (mgr.14254) 1164 : cluster [DBG] pgmap v949: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:36.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:36 smithi132 bash[18607]: cluster 2024-04-25T20:30:35.644805+0000 mgr.x (mgr.14254) 1164 : cluster [DBG] pgmap v949: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:39.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:38 smithi132 bash[18607]: cluster 2024-04-25T20:30:37.646202+0000 mgr.x (mgr.14254) 1165 : cluster [DBG] pgmap v950: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:39.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:38 smithi132 bash[18607]: cluster 2024-04-25T20:30:37.646202+0000 mgr.x (mgr.14254) 1165 : cluster [DBG] pgmap v950: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:39.224 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:30:39.224 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:30:39.976 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:30:40.977 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:30:41.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:40 smithi132 bash[18607]: audit 2024-04-25T20:30:39.221799+0000 mgr.x (mgr.14254) 1166 : audit [DBG] from='client.14838 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:30:41.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:40 smithi132 bash[18607]: audit 2024-04-25T20:30:39.221799+0000 mgr.x (mgr.14254) 1166 : audit [DBG] from='client.14838 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:30:41.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:40 smithi132 bash[18607]: cluster 2024-04-25T20:30:39.648100+0000 mgr.x (mgr.14254) 1167 : cluster [DBG] pgmap v951: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:41.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:40 smithi132 bash[18607]: cluster 2024-04-25T20:30:39.648100+0000 mgr.x (mgr.14254) 1167 : cluster [DBG] pgmap v951: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:43.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:42 smithi132 bash[18607]: cluster 2024-04-25T20:30:41.649573+0000 mgr.x (mgr.14254) 1168 : cluster [DBG] pgmap v952: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:43.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:42 smithi132 bash[18607]: cluster 2024-04-25T20:30:41.649573+0000 mgr.x (mgr.14254) 1168 : cluster [DBG] pgmap v952: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:44.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:43 smithi132 bash[18607]: audit 2024-04-25T20:30:42.910229+0000 mon.a (mon.0) 504 : audit [DBG] from='client.? 172.17.0.2:0/3845822841' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:30:44.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:43 smithi132 bash[18607]: audit 2024-04-25T20:30:42.910229+0000 mon.a (mon.0) 504 : audit [DBG] from='client.? 172.17.0.2:0/3845822841' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:30:45.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:44 smithi132 bash[18607]: cluster 2024-04-25T20:30:43.651435+0000 mgr.x (mgr.14254) 1169 : cluster [DBG] pgmap v953: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:45.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:44 smithi132 bash[18607]: cluster 2024-04-25T20:30:43.651435+0000 mgr.x (mgr.14254) 1169 : cluster [DBG] pgmap v953: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:45.741 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:30:46.998 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:46 smithi132 bash[18607]: cluster 2024-04-25T20:30:45.652897+0000 mgr.x (mgr.14254) 1170 : cluster [DBG] pgmap v954: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:46.998 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:46 smithi132 bash[18607]: cluster 2024-04-25T20:30:45.652897+0000 mgr.x (mgr.14254) 1170 : cluster [DBG] pgmap v954: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:48.736 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:30:48.736 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:30:48.964 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18607]: cluster 2024-04-25T20:30:47.654331+0000 mgr.x (mgr.14254) 1171 : cluster [DBG] pgmap v955: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:48.964 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18607]: cluster 2024-04-25T20:30:47.654331+0000 mgr.x (mgr.14254) 1171 : cluster [DBG] pgmap v955: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:48.965 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: debug 2024-04-25T20:30:48.617+0000 7f9113b3f700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig 2024-04-25T20:30:48.965 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: /usr/bin/docker: stdout 2024-04-25T20:30:48.965 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig 2024-04-25T20:30:48.965 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig 2024-04-25T20:30:48.965 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: /usr/bin/docker: stdout 2024-04-25T20:30:48.965 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig 2024-04-25T20:30:48.965 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: Reconfig daemon smb.modtest1.smithi132.jrsyig ... 2024-04-25T20:30:48.965 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: systemctl: timeout after 895 seconds 2024-04-25T20:30:48.965 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: Non-zero exit code 124 from systemctl restart ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig 2024-04-25T20:30:48.966 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: Traceback (most recent call last): 2024-04-25T20:30:48.966 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main 2024-04-25T20:30:48.966 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: return _run_code(code, main_globals, None, 2024-04-25T20:30:48.966 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: File "/usr/lib/python3.10/runpy.py", line 86, in _run_code 2024-04-25T20:30:48.966 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: exec(code, run_globals) 2024-04-25T20:30:48.966 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 5618, in 2024-04-25T20:30:48.966 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 5606, in main 2024-04-25T20:30:48.966 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 3096, in command_deploy_from 2024-04-25T20:30:48.966 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 3131, in _common_deploy 2024-04-25T20:30:48.966 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 3151, in _deploy_daemon_container 2024-04-25T20:30:48.966 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 1123, in deploy_daemon 2024-04-25T20:30:48.966 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-25T20:30:48.966 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: RuntimeError: Failed command: systemctl restart ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig 2024-04-25T20:30:48.966 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: Traceback (most recent call last): 2024-04-25T20:30:48.966 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-04-25T20:30:48.967 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-25T20:30:48.967 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2457, in _daemon_action 2024-04-25T20:30:48.967 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-25T20:30:48.967 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-04-25T20:30:48.967 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: return self.event_loop.get_result(coro, timeout) 2024-04-25T20:30:48.967 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-25T20:30:48.967 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: return future.result(timeout) 2024-04-25T20:30:48.967 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-25T20:30:48.968 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: return self.__get_result() 2024-04-25T20:30:48.968 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-25T20:30:48.968 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: raise self._exception 2024-04-25T20:30:48.968 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-04-25T20:30:48.968 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: use_current_daemon_image=reconfig, 2024-04-25T20:30:48.968 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-04-25T20:30:48.968 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-25T20:30:48.968 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig 2024-04-25T20:30:48.968 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: /usr/bin/docker: stdout 2024-04-25T20:30:48.968 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig 2024-04-25T20:30:48.968 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig 2024-04-25T20:30:48.968 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: /usr/bin/docker: stdout 2024-04-25T20:30:48.968 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig 2024-04-25T20:30:48.968 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: Reconfig daemon smb.modtest1.smithi132.jrsyig ... 2024-04-25T20:30:48.968 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: systemctl: timeout after 895 seconds 2024-04-25T20:30:48.969 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: Non-zero exit code 124 from systemctl restart ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig 2024-04-25T20:30:48.969 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: Traceback (most recent call last): 2024-04-25T20:30:48.969 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main 2024-04-25T20:30:48.969 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: return _run_code(code, main_globals, None, 2024-04-25T20:30:48.969 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: File "/usr/lib/python3.10/runpy.py", line 86, in _run_code 2024-04-25T20:30:48.969 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: exec(code, run_globals) 2024-04-25T20:30:48.969 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 5618, in 2024-04-25T20:30:48.969 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 5606, in main 2024-04-25T20:30:48.969 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 3096, in command_deploy_from 2024-04-25T20:30:48.969 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 3131, in _common_deploy 2024-04-25T20:30:48.969 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 3151, in _deploy_daemon_container 2024-04-25T20:30:48.969 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 1123, in deploy_daemon 2024-04-25T20:30:48.969 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-25T20:30:48.969 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:30:48 smithi132 bash[18860]: RuntimeError: Failed command: systemctl restart ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig 2024-04-25T20:30:49.455 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:30:50.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: cephadm 2024-04-25T20:30:48.624658+0000 mgr.x (mgr.14254) 1172 : cephadm [ERR] cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig 2024-04-25T20:30:50.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: /usr/bin/docker: stdout 2024-04-25T20:30:50.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig 2024-04-25T20:30:50.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig 2024-04-25T20:30:50.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: /usr/bin/docker: stdout 2024-04-25T20:30:50.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig 2024-04-25T20:30:50.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: Reconfig daemon smb.modtest1.smithi132.jrsyig ... 2024-04-25T20:30:50.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: systemctl: timeout after 895 seconds 2024-04-25T20:30:50.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: Non-zero exit code 124 from systemctl restart ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig 2024-04-25T20:30:50.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: Traceback (most recent call last): 2024-04-25T20:30:50.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main 2024-04-25T20:30:50.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: return _run_code(code, main_globals, None, 2024-04-25T20:30:50.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/usr/lib/python3.10/runpy.py", line 86, in _run_code 2024-04-25T20:30:50.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: exec(code, run_globals) 2024-04-25T20:30:50.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 5618, in 2024-04-25T20:30:50.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 5606, in main 2024-04-25T20:30:50.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 3096, in command_deploy_from 2024-04-25T20:30:50.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 3131, in _common_deploy 2024-04-25T20:30:50.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 3151, in _deploy_daemon_container 2024-04-25T20:30:50.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 1123, in deploy_daemon 2024-04-25T20:30:50.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-25T20:30:50.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: RuntimeError: Failed command: systemctl restart ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig 2024-04-25T20:30:50.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: Traceback (most recent call last): 2024-04-25T20:30:50.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-04-25T20:30:50.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-25T20:30:50.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2457, in _daemon_action 2024-04-25T20:30:50.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-25T20:30:50.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-04-25T20:30:50.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: return self.event_loop.get_result(coro, timeout) 2024-04-25T20:30:50.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-25T20:30:50.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: return future.result(timeout) 2024-04-25T20:30:50.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-25T20:30:50.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: return self.__get_result() 2024-04-25T20:30:50.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-25T20:30:50.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: raise self._exception 2024-04-25T20:30:50.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-04-25T20:30:50.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: use_current_daemon_image=reconfig, 2024-04-25T20:30:50.275 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-04-25T20:30:50.276 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-25T20:30:50.276 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig 2024-04-25T20:30:50.276 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: /usr/bin/docker: stdout 2024-04-25T20:30:50.276 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig 2024-04-25T20:30:50.276 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig 2024-04-25T20:30:50.276 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: /usr/bin/docker: stdout 2024-04-25T20:30:50.276 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig 2024-04-25T20:30:50.276 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: Reconfig daemon smb.modtest1.smithi132.jrsyig ... 2024-04-25T20:30:50.276 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: systemctl: timeout after 895 seconds 2024-04-25T20:30:50.276 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: Non-zero exit code 124 from systemctl restart ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig 2024-04-25T20:30:50.276 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: Traceback (most recent call last): 2024-04-25T20:30:50.276 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main 2024-04-25T20:30:50.276 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: return _run_code(code, main_globals, None, 2024-04-25T20:30:50.276 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/usr/lib/python3.10/runpy.py", line 86, in _run_code 2024-04-25T20:30:50.276 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: exec(code, run_globals) 2024-04-25T20:30:50.277 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 5618, in 2024-04-25T20:30:50.277 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 5606, in main 2024-04-25T20:30:50.277 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 3096, in command_deploy_from 2024-04-25T20:30:50.277 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 3131, in _common_deploy 2024-04-25T20:30:50.277 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 3151, in _deploy_daemon_container 2024-04-25T20:30:50.277 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 1123, in deploy_daemon 2024-04-25T20:30:50.277 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-25T20:30:50.277 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: RuntimeError: Failed command: systemctl restart ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig 2024-04-25T20:30:50.277 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: cephadm 2024-04-25T20:30:48.624658+0000 mgr.x (mgr.14254) 1172 : cephadm [ERR] cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig 2024-04-25T20:30:50.277 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: /usr/bin/docker: stdout 2024-04-25T20:30:50.277 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig 2024-04-25T20:30:50.277 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig 2024-04-25T20:30:50.277 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: /usr/bin/docker: stdout 2024-04-25T20:30:50.277 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig 2024-04-25T20:30:50.277 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: Reconfig daemon smb.modtest1.smithi132.jrsyig ... 2024-04-25T20:30:50.278 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: systemctl: timeout after 895 seconds 2024-04-25T20:30:50.278 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: Non-zero exit code 124 from systemctl restart ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig 2024-04-25T20:30:50.278 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: Traceback (most recent call last): 2024-04-25T20:30:50.278 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main 2024-04-25T20:30:50.278 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: return _run_code(code, main_globals, None, 2024-04-25T20:30:50.278 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/usr/lib/python3.10/runpy.py", line 86, in _run_code 2024-04-25T20:30:50.278 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: exec(code, run_globals) 2024-04-25T20:30:50.278 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 5618, in 2024-04-25T20:30:50.278 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 5606, in main 2024-04-25T20:30:50.278 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 3096, in command_deploy_from 2024-04-25T20:30:50.278 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 3131, in _common_deploy 2024-04-25T20:30:50.278 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 3151, in _deploy_daemon_container 2024-04-25T20:30:50.278 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 1123, in deploy_daemon 2024-04-25T20:30:50.278 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-25T20:30:50.278 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: RuntimeError: Failed command: systemctl restart ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig 2024-04-25T20:30:50.278 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: Traceback (most recent call last): 2024-04-25T20:30:50.279 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-04-25T20:30:50.279 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-25T20:30:50.279 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2457, in _daemon_action 2024-04-25T20:30:50.279 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-25T20:30:50.279 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-04-25T20:30:50.279 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: return self.event_loop.get_result(coro, timeout) 2024-04-25T20:30:50.279 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-25T20:30:50.279 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: return future.result(timeout) 2024-04-25T20:30:50.279 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-25T20:30:50.279 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: return self.__get_result() 2024-04-25T20:30:50.279 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-25T20:30:50.279 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: raise self._exception 2024-04-25T20:30:50.279 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-04-25T20:30:50.279 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: use_current_daemon_image=reconfig, 2024-04-25T20:30:50.279 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-04-25T20:30:50.280 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-25T20:30:50.280 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig 2024-04-25T20:30:50.280 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: /usr/bin/docker: stdout 2024-04-25T20:30:50.280 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig 2024-04-25T20:30:50.280 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig 2024-04-25T20:30:50.280 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: /usr/bin/docker: stdout 2024-04-25T20:30:50.280 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig 2024-04-25T20:30:50.280 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: Reconfig daemon smb.modtest1.smithi132.jrsyig ... 2024-04-25T20:30:50.280 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: systemctl: timeout after 895 seconds 2024-04-25T20:30:50.280 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: Non-zero exit code 124 from systemctl restart ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig 2024-04-25T20:30:50.280 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: Traceback (most recent call last): 2024-04-25T20:30:50.280 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main 2024-04-25T20:30:50.280 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: return _run_code(code, main_globals, None, 2024-04-25T20:30:50.280 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/usr/lib/python3.10/runpy.py", line 86, in _run_code 2024-04-25T20:30:50.280 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: exec(code, run_globals) 2024-04-25T20:30:50.281 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 5618, in 2024-04-25T20:30:50.281 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 5606, in main 2024-04-25T20:30:50.281 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 3096, in command_deploy_from 2024-04-25T20:30:50.281 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 3131, in _common_deploy 2024-04-25T20:30:50.281 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 3151, in _deploy_daemon_container 2024-04-25T20:30:50.281 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py", line 1123, in deploy_daemon 2024-04-25T20:30:50.281 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: File "/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py", line 310, in call_throws 2024-04-25T20:30:50.281 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: RuntimeError: Failed command: systemctl restart ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig 2024-04-25T20:30:50.281 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: audit 2024-04-25T20:30:48.733504+0000 mgr.x (mgr.14254) 1173 : audit [DBG] from='client.14842 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:30:50.281 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:49 smithi132 bash[18607]: audit 2024-04-25T20:30:48.733504+0000 mgr.x (mgr.14254) 1173 : audit [DBG] from='client.14842 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:30:50.456 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:30:51.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:50 smithi132 bash[18607]: cluster 2024-04-25T20:30:49.656332+0000 mgr.x (mgr.14254) 1174 : cluster [DBG] pgmap v956: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:51.272 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:50 smithi132 bash[18607]: cluster 2024-04-25T20:30:49.656332+0000 mgr.x (mgr.14254) 1174 : cluster [DBG] pgmap v956: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:53.218 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:52 smithi132 bash[18607]: cluster 2024-04-25T20:30:51.657953+0000 mgr.x (mgr.14254) 1175 : cluster [DBG] pgmap v957: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:53.218 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:52 smithi132 bash[18607]: cluster 2024-04-25T20:30:51.657953+0000 mgr.x (mgr.14254) 1175 : cluster [DBG] pgmap v957: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:55.221 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:30:55.242 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:54 smithi132 bash[18607]: cluster 2024-04-25T20:30:53.659871+0000 mgr.x (mgr.14254) 1176 : cluster [DBG] pgmap v958: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:55.242 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:54 smithi132 bash[18607]: cluster 2024-04-25T20:30:53.659871+0000 mgr.x (mgr.14254) 1176 : cluster [DBG] pgmap v958: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:56.178 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:55 smithi132 bash[18607]: cluster 2024-04-25T20:30:55.661384+0000 mgr.x (mgr.14254) 1177 : cluster [DBG] pgmap v959: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:56.178 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:55 smithi132 bash[18607]: cluster 2024-04-25T20:30:55.661384+0000 mgr.x (mgr.14254) 1177 : cluster [DBG] pgmap v959: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:58.071 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:30:58.071 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:30:58.776 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:30:59.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:58 smithi132 bash[18607]: cluster 2024-04-25T20:30:57.662892+0000 mgr.x (mgr.14254) 1178 : cluster [DBG] pgmap v960: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:59.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:58 smithi132 bash[18607]: cluster 2024-04-25T20:30:57.662892+0000 mgr.x (mgr.14254) 1178 : cluster [DBG] pgmap v960: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:30:59.777 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:31:00.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:59 smithi132 bash[18607]: audit 2024-04-25T20:30:58.069456+0000 mgr.x (mgr.14254) 1179 : audit [DBG] from='client.14844 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:31:00.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:30:59 smithi132 bash[18607]: audit 2024-04-25T20:30:58.069456+0000 mgr.x (mgr.14254) 1179 : audit [DBG] from='client.14844 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:31:01.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:00 smithi132 bash[18607]: cluster 2024-04-25T20:30:59.664875+0000 mgr.x (mgr.14254) 1180 : cluster [DBG] pgmap v961: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:01.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:00 smithi132 bash[18607]: cluster 2024-04-25T20:30:59.664875+0000 mgr.x (mgr.14254) 1180 : cluster [DBG] pgmap v961: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:03.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:02 smithi132 bash[18607]: cluster 2024-04-25T20:31:01.666281+0000 mgr.x (mgr.14254) 1181 : cluster [DBG] pgmap v962: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:03.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:02 smithi132 bash[18607]: cluster 2024-04-25T20:31:01.666281+0000 mgr.x (mgr.14254) 1181 : cluster [DBG] pgmap v962: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:04.535 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:31:05.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:04 smithi132 bash[18607]: cluster 2024-04-25T20:31:03.668144+0000 mgr.x (mgr.14254) 1182 : cluster [DBG] pgmap v963: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:05.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:04 smithi132 bash[18607]: cluster 2024-04-25T20:31:03.668144+0000 mgr.x (mgr.14254) 1182 : cluster [DBG] pgmap v963: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:07.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:06 smithi132 bash[18607]: cluster 2024-04-25T20:31:05.669616+0000 mgr.x (mgr.14254) 1183 : cluster [DBG] pgmap v964: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:07.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:06 smithi132 bash[18607]: cluster 2024-04-25T20:31:05.669616+0000 mgr.x (mgr.14254) 1183 : cluster [DBG] pgmap v964: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:07.274 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:31:07.274 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:31:07.975 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:31:08.977 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:31:08.988 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:08 smithi132 bash[18607]: audit 2024-04-25T20:31:07.271688+0000 mgr.x (mgr.14254) 1184 : audit [DBG] from='client.14846 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:31:08.989 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:08 smithi132 bash[18607]: audit 2024-04-25T20:31:07.271688+0000 mgr.x (mgr.14254) 1184 : audit [DBG] from='client.14846 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:31:08.989 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:08 smithi132 bash[18607]: cluster 2024-04-25T20:31:07.671038+0000 mgr.x (mgr.14254) 1185 : cluster [DBG] pgmap v965: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:08.989 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:08 smithi132 bash[18607]: cluster 2024-04-25T20:31:07.671038+0000 mgr.x (mgr.14254) 1185 : cluster [DBG] pgmap v965: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:11.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:10 smithi132 bash[18607]: cluster 2024-04-25T20:31:09.672910+0000 mgr.x (mgr.14254) 1186 : cluster [DBG] pgmap v966: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:11.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:10 smithi132 bash[18607]: cluster 2024-04-25T20:31:09.672910+0000 mgr.x (mgr.14254) 1186 : cluster [DBG] pgmap v966: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:13.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:12 smithi132 bash[18607]: cluster 2024-04-25T20:31:11.674385+0000 mgr.x (mgr.14254) 1187 : cluster [DBG] pgmap v967: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:13.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:12 smithi132 bash[18607]: cluster 2024-04-25T20:31:11.674385+0000 mgr.x (mgr.14254) 1187 : cluster [DBG] pgmap v967: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:13.740 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:31:14.843 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:14 smithi132 bash[18607]: cluster 2024-04-25T20:31:13.676251+0000 mgr.x (mgr.14254) 1188 : cluster [DBG] pgmap v968: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:14.843 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:14 smithi132 bash[18607]: cluster 2024-04-25T20:31:13.676251+0000 mgr.x (mgr.14254) 1188 : cluster [DBG] pgmap v968: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:16.425 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:31:16.425 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:31:17.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:16 smithi132 bash[18607]: cluster 2024-04-25T20:31:15.677762+0000 mgr.x (mgr.14254) 1189 : cluster [DBG] pgmap v969: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:17.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:16 smithi132 bash[18607]: cluster 2024-04-25T20:31:15.677762+0000 mgr.x (mgr.14254) 1189 : cluster [DBG] pgmap v969: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:17.174 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:31:18.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:17 smithi132 bash[18607]: audit 2024-04-25T20:31:16.426024+0000 mgr.x (mgr.14254) 1190 : audit [DBG] from='client.14848 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:31:18.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:17 smithi132 bash[18607]: audit 2024-04-25T20:31:16.426024+0000 mgr.x (mgr.14254) 1190 : audit [DBG] from='client.14848 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:31:18.175 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:31:19.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:18 smithi132 bash[18607]: cluster 2024-04-25T20:31:17.679275+0000 mgr.x (mgr.14254) 1191 : cluster [DBG] pgmap v970: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:19.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:18 smithi132 bash[18607]: cluster 2024-04-25T20:31:17.679275+0000 mgr.x (mgr.14254) 1191 : cluster [DBG] pgmap v970: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:21.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:20 smithi132 bash[18607]: cluster 2024-04-25T20:31:19.681187+0000 mgr.x (mgr.14254) 1192 : cluster [DBG] pgmap v971: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:21.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:20 smithi132 bash[18607]: cluster 2024-04-25T20:31:19.681187+0000 mgr.x (mgr.14254) 1192 : cluster [DBG] pgmap v971: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:22.933 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:31:23.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:22 smithi132 bash[18607]: cluster 2024-04-25T20:31:21.682630+0000 mgr.x (mgr.14254) 1193 : cluster [DBG] pgmap v972: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:23.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:22 smithi132 bash[18607]: cluster 2024-04-25T20:31:21.682630+0000 mgr.x (mgr.14254) 1193 : cluster [DBG] pgmap v972: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:25.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:24 smithi132 bash[18607]: cluster 2024-04-25T20:31:23.684531+0000 mgr.x (mgr.14254) 1194 : cluster [DBG] pgmap v973: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:25.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:24 smithi132 bash[18607]: cluster 2024-04-25T20:31:23.684531+0000 mgr.x (mgr.14254) 1194 : cluster [DBG] pgmap v973: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:25.834 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:31:25.834 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:31:26.534 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:31:27.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:26 smithi132 bash[18607]: cluster 2024-04-25T20:31:25.685921+0000 mgr.x (mgr.14254) 1195 : cluster [DBG] pgmap v974: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:27.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:26 smithi132 bash[18607]: cluster 2024-04-25T20:31:25.685921+0000 mgr.x (mgr.14254) 1195 : cluster [DBG] pgmap v974: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:27.535 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:31:28.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:27 smithi132 bash[18607]: audit 2024-04-25T20:31:25.833509+0000 mgr.x (mgr.14254) 1196 : audit [DBG] from='client.14850 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:31:28.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:27 smithi132 bash[18607]: audit 2024-04-25T20:31:25.833509+0000 mgr.x (mgr.14254) 1196 : audit [DBG] from='client.14850 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:31:29.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:28 smithi132 bash[18607]: cluster 2024-04-25T20:31:27.687310+0000 mgr.x (mgr.14254) 1197 : cluster [DBG] pgmap v975: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:29.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:28 smithi132 bash[18607]: cluster 2024-04-25T20:31:27.687310+0000 mgr.x (mgr.14254) 1197 : cluster [DBG] pgmap v975: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:31.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:30 smithi132 bash[18607]: cluster 2024-04-25T20:31:29.689331+0000 mgr.x (mgr.14254) 1198 : cluster [DBG] pgmap v976: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:31.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:30 smithi132 bash[18607]: cluster 2024-04-25T20:31:29.689331+0000 mgr.x (mgr.14254) 1198 : cluster [DBG] pgmap v976: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:32.299 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:31:33.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:32 smithi132 bash[18607]: cluster 2024-04-25T20:31:31.690908+0000 mgr.x (mgr.14254) 1199 : cluster [DBG] pgmap v977: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:33.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:32 smithi132 bash[18607]: cluster 2024-04-25T20:31:31.690908+0000 mgr.x (mgr.14254) 1199 : cluster [DBG] pgmap v977: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:34.988 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:31:34.988 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:31:35.001 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:34 smithi132 bash[18607]: cluster 2024-04-25T20:31:33.692020+0000 mgr.x (mgr.14254) 1200 : cluster [DBG] pgmap v978: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:35.002 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:34 smithi132 bash[18607]: cluster 2024-04-25T20:31:33.692020+0000 mgr.x (mgr.14254) 1200 : cluster [DBG] pgmap v978: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:35.655 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:31:36.657 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:31:37.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:36 smithi132 bash[18607]: audit 2024-04-25T20:31:34.986019+0000 mgr.x (mgr.14254) 1201 : audit [DBG] from='client.14852 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:31:37.088 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:36 smithi132 bash[18607]: audit 2024-04-25T20:31:34.986019+0000 mgr.x (mgr.14254) 1201 : audit [DBG] from='client.14852 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:31:37.088 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:36 smithi132 bash[18607]: cluster 2024-04-25T20:31:35.693476+0000 mgr.x (mgr.14254) 1202 : cluster [DBG] pgmap v979: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:37.089 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:36 smithi132 bash[18607]: cluster 2024-04-25T20:31:35.693476+0000 mgr.x (mgr.14254) 1202 : cluster [DBG] pgmap v979: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:39.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:38 smithi132 bash[18607]: cluster 2024-04-25T20:31:37.694898+0000 mgr.x (mgr.14254) 1203 : cluster [DBG] pgmap v980: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:39.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:38 smithi132 bash[18607]: cluster 2024-04-25T20:31:37.694898+0000 mgr.x (mgr.14254) 1203 : cluster [DBG] pgmap v980: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:41.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:40 smithi132 bash[18607]: cluster 2024-04-25T20:31:39.696802+0000 mgr.x (mgr.14254) 1204 : cluster [DBG] pgmap v981: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:41.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:40 smithi132 bash[18607]: cluster 2024-04-25T20:31:39.696802+0000 mgr.x (mgr.14254) 1204 : cluster [DBG] pgmap v981: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:41.424 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:31:43.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:42 smithi132 bash[18607]: cluster 2024-04-25T20:31:41.698238+0000 mgr.x (mgr.14254) 1205 : cluster [DBG] pgmap v982: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:43.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:42 smithi132 bash[18607]: cluster 2024-04-25T20:31:41.698238+0000 mgr.x (mgr.14254) 1205 : cluster [DBG] pgmap v982: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:44.174 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:31:44.174 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:31:44.891 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:31:45.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:44 smithi132 bash[18607]: cluster 2024-04-25T20:31:43.700144+0000 mgr.x (mgr.14254) 1206 : cluster [DBG] pgmap v983: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:45.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:44 smithi132 bash[18607]: cluster 2024-04-25T20:31:43.700144+0000 mgr.x (mgr.14254) 1206 : cluster [DBG] pgmap v983: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:45.892 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:31:46.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:45 smithi132 bash[18607]: audit 2024-04-25T20:31:44.174894+0000 mgr.x (mgr.14254) 1207 : audit [DBG] from='client.14854 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:31:46.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:45 smithi132 bash[18607]: audit 2024-04-25T20:31:44.174894+0000 mgr.x (mgr.14254) 1207 : audit [DBG] from='client.14854 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:31:47.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:46 smithi132 bash[18607]: cluster 2024-04-25T20:31:45.701635+0000 mgr.x (mgr.14254) 1208 : cluster [DBG] pgmap v984: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:47.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:46 smithi132 bash[18607]: cluster 2024-04-25T20:31:45.701635+0000 mgr.x (mgr.14254) 1208 : cluster [DBG] pgmap v984: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:49.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:48 smithi132 bash[18607]: cluster 2024-04-25T20:31:47.702997+0000 mgr.x (mgr.14254) 1209 : cluster [DBG] pgmap v985: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:49.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:48 smithi132 bash[18607]: cluster 2024-04-25T20:31:47.702997+0000 mgr.x (mgr.14254) 1209 : cluster [DBG] pgmap v985: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:49.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:48 smithi132 bash[18607]: audit 2024-04-25T20:31:48.626053+0000 mon.a (mon.0) 505 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T20:31:49.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:48 smithi132 bash[18607]: audit 2024-04-25T20:31:48.626053+0000 mon.a (mon.0) 505 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T20:31:50.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:50 smithi132 bash[18607]: audit 2024-04-25T20:31:49.097590+0000 mon.a (mon.0) 506 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T20:31:50.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:50 smithi132 bash[18607]: audit 2024-04-25T20:31:49.097590+0000 mon.a (mon.0) 506 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T20:31:50.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:50 smithi132 bash[18607]: audit 2024-04-25T20:31:49.103108+0000 mon.a (mon.0) 507 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T20:31:50.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:50 smithi132 bash[18607]: audit 2024-04-25T20:31:49.103108+0000 mon.a (mon.0) 507 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T20:31:50.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:50 smithi132 bash[18607]: cluster 2024-04-25T20:31:49.704842+0000 mgr.x (mgr.14254) 1210 : cluster [DBG] pgmap v986: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:50.522 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:50 smithi132 bash[18607]: cluster 2024-04-25T20:31:49.704842+0000 mgr.x (mgr.14254) 1210 : cluster [DBG] pgmap v986: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:50.649 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:31:53.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:52 smithi132 bash[18607]: cluster 2024-04-25T20:31:51.705595+0000 mgr.x (mgr.14254) 1211 : cluster [DBG] pgmap v987: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:53.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:52 smithi132 bash[18607]: cluster 2024-04-25T20:31:51.705595+0000 mgr.x (mgr.14254) 1211 : cluster [DBG] pgmap v987: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:53.346 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:31:53.347 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:15:52.307364Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:15:52.306470Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:15:52.306137Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:15:52.306772Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:15:52.307761Z", "running": 0, "size": 1}}] 2024-04-25T20:31:54.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:53 smithi132 bash[18607]: audit 2024-04-25T20:31:53.293648+0000 mon.a (mon.0) 508 : audit [DBG] from='client.? 172.17.0.2:0/2618520323' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:31:54.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:53 smithi132 bash[18607]: audit 2024-04-25T20:31:53.293648+0000 mon.a (mon.0) 508 : audit [DBG] from='client.? 172.17.0.2:0/2618520323' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:31:54.111 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:31:55.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:54 smithi132 bash[18607]: audit 2024-04-25T20:31:53.348141+0000 mgr.x (mgr.14254) 1212 : audit [DBG] from='client.14856 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:31:55.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:54 smithi132 bash[18607]: audit 2024-04-25T20:31:53.348141+0000 mgr.x (mgr.14254) 1212 : audit [DBG] from='client.14856 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:31:55.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:54 smithi132 bash[18607]: cluster 2024-04-25T20:31:53.707541+0000 mgr.x (mgr.14254) 1213 : cluster [DBG] pgmap v988: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:55.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:54 smithi132 bash[18607]: cluster 2024-04-25T20:31:53.707541+0000 mgr.x (mgr.14254) 1213 : cluster [DBG] pgmap v988: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:55.112 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:31:56.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:55 smithi132 bash[18607]: audit 2024-04-25T20:31:54.937267+0000 mon.a (mon.0) 509 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T20:31:56.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:55 smithi132 bash[18607]: audit 2024-04-25T20:31:54.937267+0000 mon.a (mon.0) 509 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T20:31:56.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:55 smithi132 bash[18607]: audit 2024-04-25T20:31:54.940594+0000 mon.a (mon.0) 510 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T20:31:56.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:55 smithi132 bash[18607]: audit 2024-04-25T20:31:54.940594+0000 mon.a (mon.0) 510 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T20:31:56.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:55 smithi132 bash[18607]: cluster 2024-04-25T20:31:55.709100+0000 mgr.x (mgr.14254) 1214 : cluster [DBG] pgmap v989: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:56.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:55 smithi132 bash[18607]: cluster 2024-04-25T20:31:55.709100+0000 mgr.x (mgr.14254) 1214 : cluster [DBG] pgmap v989: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:56.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:55 smithi132 bash[18607]: audit 2024-04-25T20:31:55.881007+0000 mon.a (mon.0) 511 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T20:31:56.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:55 smithi132 bash[18607]: audit 2024-04-25T20:31:55.881007+0000 mon.a (mon.0) 511 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T20:31:56.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:55 smithi132 bash[18607]: audit 2024-04-25T20:31:55.888731+0000 mon.a (mon.0) 512 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T20:31:56.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:55 smithi132 bash[18607]: audit 2024-04-25T20:31:55.888731+0000 mon.a (mon.0) 512 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T20:31:59.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:58 smithi132 bash[18607]: cluster 2024-04-25T20:31:57.710533+0000 mgr.x (mgr.14254) 1215 : cluster [DBG] pgmap v990: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:59.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:31:58 smithi132 bash[18607]: cluster 2024-04-25T20:31:57.710533+0000 mgr.x (mgr.14254) 1215 : cluster [DBG] pgmap v990: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:31:59.856 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:32:01.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:00 smithi132 bash[18607]: cluster 2024-04-25T20:31:59.712362+0000 mgr.x (mgr.14254) 1216 : cluster [DBG] pgmap v991: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:01.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:00 smithi132 bash[18607]: cluster 2024-04-25T20:31:59.712362+0000 mgr.x (mgr.14254) 1216 : cluster [DBG] pgmap v991: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:03.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:02 smithi132 bash[18607]: cluster 2024-04-25T20:32:01.713789+0000 mgr.x (mgr.14254) 1217 : cluster [DBG] pgmap v992: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:03.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:02 smithi132 bash[18607]: cluster 2024-04-25T20:32:01.713789+0000 mgr.x (mgr.14254) 1217 : cluster [DBG] pgmap v992: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:03.693 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:32:03.693 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:32:04.502 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:32:05.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:04 smithi132 bash[18607]: audit 2024-04-25T20:32:03.694484+0000 mgr.x (mgr.14254) 1218 : audit [DBG] from='client.14860 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:32:05.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:04 smithi132 bash[18607]: audit 2024-04-25T20:32:03.694484+0000 mgr.x (mgr.14254) 1218 : audit [DBG] from='client.14860 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:32:05.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:04 smithi132 bash[18607]: cluster 2024-04-25T20:32:03.714657+0000 mgr.x (mgr.14254) 1219 : cluster [DBG] pgmap v993: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:05.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:04 smithi132 bash[18607]: cluster 2024-04-25T20:32:03.714657+0000 mgr.x (mgr.14254) 1219 : cluster [DBG] pgmap v993: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:05.504 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:32:06.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:06 smithi132 bash[18607]: audit 2024-04-25T20:32:05.333928+0000 mon.a (mon.0) 513 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T20:32:06.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:06 smithi132 bash[18607]: audit 2024-04-25T20:32:05.333928+0000 mon.a (mon.0) 513 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T20:32:06.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:06 smithi132 bash[18607]: audit 2024-04-25T20:32:05.342124+0000 mon.a (mon.0) 514 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T20:32:06.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:06 smithi132 bash[18607]: audit 2024-04-25T20:32:05.342124+0000 mon.a (mon.0) 514 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T20:32:06.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:06 smithi132 bash[18607]: audit 2024-04-25T20:32:05.344953+0000 mon.a (mon.0) 515 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T20:32:06.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:06 smithi132 bash[18607]: audit 2024-04-25T20:32:05.344953+0000 mon.a (mon.0) 515 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T20:32:06.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:06 smithi132 bash[18607]: audit 2024-04-25T20:32:05.346719+0000 mon.a (mon.0) 516 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T20:32:06.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:06 smithi132 bash[18607]: audit 2024-04-25T20:32:05.346719+0000 mon.a (mon.0) 516 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T20:32:06.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:06 smithi132 bash[18607]: cluster 2024-04-25T20:32:05.350597+0000 mgr.x (mgr.14254) 1220 : cluster [DBG] pgmap v994: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:06.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:06 smithi132 bash[18607]: cluster 2024-04-25T20:32:05.350597+0000 mgr.x (mgr.14254) 1220 : cluster [DBG] pgmap v994: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:06.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:06 smithi132 bash[18607]: audit 2024-04-25T20:32:05.355712+0000 mon.a (mon.0) 517 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T20:32:06.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:06 smithi132 bash[18607]: audit 2024-04-25T20:32:05.355712+0000 mon.a (mon.0) 517 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' 2024-04-25T20:32:06.772 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:06 smithi132 bash[18607]: cephadm 2024-04-25T20:32:05.394668+0000 mgr.x (mgr.14254) 1221 : cephadm [INF] Reconfiguring smb.modtest1.smithi132.jrsyig (unknown last config time)... 2024-04-25T20:32:06.773 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:06 smithi132 bash[18607]: cephadm 2024-04-25T20:32:05.394668+0000 mgr.x (mgr.14254) 1221 : cephadm [INF] Reconfiguring smb.modtest1.smithi132.jrsyig (unknown last config time)... 2024-04-25T20:32:06.773 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:06 smithi132 bash[18607]: audit 2024-04-25T20:32:05.395623+0000 mon.a (mon.0) 518 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.smb.config.modtest1.smithi132.jrsyig", "caps": ["mon", "allow r, allow command \"config-key get\" with \"key\" prefix \"smb/config/modtest1/\"", "osd", "allow r pool=.smb"]}]: dispatch 2024-04-25T20:32:06.773 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:06 smithi132 bash[18607]: audit 2024-04-25T20:32:05.395623+0000 mon.a (mon.0) 518 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.smb.config.modtest1.smithi132.jrsyig", "caps": ["mon", "allow r, allow command \"config-key get\" with \"key\" prefix \"smb/config/modtest1/\"", "osd", "allow r pool=.smb"]}]: dispatch 2024-04-25T20:32:06.773 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:06 smithi132 bash[18607]: audit 2024-04-25T20:32:05.397316+0000 mon.a (mon.0) 519 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.smb.fs.cluster.modtest1"}]: dispatch 2024-04-25T20:32:06.773 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:06 smithi132 bash[18607]: audit 2024-04-25T20:32:05.397316+0000 mon.a (mon.0) 519 : audit [INF] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.smb.fs.cluster.modtest1"}]: dispatch 2024-04-25T20:32:06.773 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:06 smithi132 bash[18607]: audit 2024-04-25T20:32:05.398459+0000 mon.a (mon.0) 520 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T20:32:06.773 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:06 smithi132 bash[18607]: audit 2024-04-25T20:32:05.398459+0000 mon.a (mon.0) 520 : audit [DBG] from='mgr.14254 172.21.15.132:0/1898562375' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T20:32:06.773 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:06 smithi132 bash[18607]: cephadm 2024-04-25T20:32:05.400082+0000 mgr.x (mgr.14254) 1222 : cephadm [INF] Reconfiguring daemon smb.modtest1.smithi132.jrsyig on smithi132 2024-04-25T20:32:06.773 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:06 smithi132 bash[18607]: cephadm 2024-04-25T20:32:05.400082+0000 mgr.x (mgr.14254) 1222 : cephadm [INF] Reconfiguring daemon smb.modtest1.smithi132.jrsyig on smithi132 2024-04-25T20:32:07.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:07 smithi132 bash[18607]: cluster 2024-04-25T20:32:06.338885+0000 mon.a (mon.0) 521 : cluster [INF] Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-25T20:32:07.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:07 smithi132 bash[18607]: cluster 2024-04-25T20:32:06.338885+0000 mon.a (mon.0) 521 : cluster [INF] Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-25T20:32:08.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:08 smithi132 bash[18607]: cluster 2024-04-25T20:32:07.352038+0000 mgr.x (mgr.14254) 1223 : cluster [DBG] pgmap v995: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:08.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:08 smithi132 bash[18607]: cluster 2024-04-25T20:32:07.352038+0000 mgr.x (mgr.14254) 1223 : cluster [DBG] pgmap v995: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:10.261 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:32:10.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:10 smithi132 bash[18607]: cluster 2024-04-25T20:32:09.353919+0000 mgr.x (mgr.14254) 1224 : cluster [DBG] pgmap v996: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:10.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:10 smithi132 bash[18607]: cluster 2024-04-25T20:32:09.353919+0000 mgr.x (mgr.14254) 1224 : cluster [DBG] pgmap v996: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:12.678 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:12 smithi132 bash[18607]: cluster 2024-04-25T20:32:11.354763+0000 mgr.x (mgr.14254) 1225 : cluster [DBG] pgmap v997: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:12.678 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:12 smithi132 bash[18607]: cluster 2024-04-25T20:32:11.354763+0000 mgr.x (mgr.14254) 1225 : cluster [DBG] pgmap v997: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:13.076 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:32:13.077 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:32:13.808 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:32:14.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:14 smithi132 bash[18607]: audit 2024-04-25T20:32:13.074731+0000 mgr.x (mgr.14254) 1226 : audit [DBG] from='client.14862 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:32:14.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:14 smithi132 bash[18607]: audit 2024-04-25T20:32:13.074731+0000 mgr.x (mgr.14254) 1226 : audit [DBG] from='client.14862 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:32:14.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:14 smithi132 bash[18607]: cluster 2024-04-25T20:32:13.356556+0000 mgr.x (mgr.14254) 1227 : cluster [DBG] pgmap v998: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:14.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:14 smithi132 bash[18607]: cluster 2024-04-25T20:32:13.356556+0000 mgr.x (mgr.14254) 1227 : cluster [DBG] pgmap v998: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:14.810 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:32:16.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:16 smithi132 bash[18607]: cluster 2024-04-25T20:32:15.357989+0000 mgr.x (mgr.14254) 1228 : cluster [DBG] pgmap v999: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:16.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:16 smithi132 bash[18607]: cluster 2024-04-25T20:32:15.357989+0000 mgr.x (mgr.14254) 1228 : cluster [DBG] pgmap v999: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:18.773 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:18 smithi132 bash[18607]: cluster 2024-04-25T20:32:17.359313+0000 mgr.x (mgr.14254) 1229 : cluster [DBG] pgmap v1000: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:18.773 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:18 smithi132 bash[18607]: cluster 2024-04-25T20:32:17.359313+0000 mgr.x (mgr.14254) 1229 : cluster [DBG] pgmap v1000: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:19.571 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:32:20.699 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:20 smithi132 bash[18607]: cluster 2024-04-25T20:32:19.361187+0000 mgr.x (mgr.14254) 1230 : cluster [DBG] pgmap v1001: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:20.699 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:20 smithi132 bash[18607]: cluster 2024-04-25T20:32:19.361187+0000 mgr.x (mgr.14254) 1230 : cluster [DBG] pgmap v1001: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:22.361 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:32:22.361 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:32:22.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:22 smithi132 bash[18607]: cluster 2024-04-25T20:32:21.362614+0000 mgr.x (mgr.14254) 1231 : cluster [DBG] pgmap v1002: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:22.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:22 smithi132 bash[18607]: cluster 2024-04-25T20:32:21.362614+0000 mgr.x (mgr.14254) 1231 : cluster [DBG] pgmap v1002: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:23.070 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:32:23.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:23 smithi132 bash[18607]: audit 2024-04-25T20:32:22.362533+0000 mgr.x (mgr.14254) 1232 : audit [DBG] from='client.14864 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:32:23.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:23 smithi132 bash[18607]: audit 2024-04-25T20:32:22.362533+0000 mgr.x (mgr.14254) 1232 : audit [DBG] from='client.14864 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:32:24.071 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:32:24.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:24 smithi132 bash[18607]: cluster 2024-04-25T20:32:23.364502+0000 mgr.x (mgr.14254) 1233 : cluster [DBG] pgmap v1003: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:24.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:24 smithi132 bash[18607]: cluster 2024-04-25T20:32:23.364502+0000 mgr.x (mgr.14254) 1233 : cluster [DBG] pgmap v1003: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:26.773 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:26 smithi132 bash[18607]: cluster 2024-04-25T20:32:25.365959+0000 mgr.x (mgr.14254) 1234 : cluster [DBG] pgmap v1004: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:26.773 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:26 smithi132 bash[18607]: cluster 2024-04-25T20:32:25.365959+0000 mgr.x (mgr.14254) 1234 : cluster [DBG] pgmap v1004: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:28.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:28 smithi132 bash[18607]: cluster 2024-04-25T20:32:27.367378+0000 mgr.x (mgr.14254) 1235 : cluster [DBG] pgmap v1005: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:28.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:28 smithi132 bash[18607]: cluster 2024-04-25T20:32:27.367378+0000 mgr.x (mgr.14254) 1235 : cluster [DBG] pgmap v1005: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:28.823 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:32:30.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:30 smithi132 bash[18607]: cluster 2024-04-25T20:32:29.369262+0000 mgr.x (mgr.14254) 1236 : cluster [DBG] pgmap v1006: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:30.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:30 smithi132 bash[18607]: cluster 2024-04-25T20:32:29.369262+0000 mgr.x (mgr.14254) 1236 : cluster [DBG] pgmap v1006: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:31.738 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:32:31.739 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:32:32.423 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:32:32.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:32 smithi132 bash[18607]: cluster 2024-04-25T20:32:31.369974+0000 mgr.x (mgr.14254) 1237 : cluster [DBG] pgmap v1007: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:32.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:32 smithi132 bash[18607]: cluster 2024-04-25T20:32:31.369974+0000 mgr.x (mgr.14254) 1237 : cluster [DBG] pgmap v1007: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:32.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:32 smithi132 bash[18607]: audit 2024-04-25T20:32:31.740181+0000 mgr.x (mgr.14254) 1238 : audit [DBG] from='client.14866 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:32:32.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:32 smithi132 bash[18607]: audit 2024-04-25T20:32:31.740181+0000 mgr.x (mgr.14254) 1238 : audit [DBG] from='client.14866 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:32:33.424 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:32:34.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:34 smithi132 bash[18607]: cluster 2024-04-25T20:32:33.371720+0000 mgr.x (mgr.14254) 1239 : cluster [DBG] pgmap v1008: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:34.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:34 smithi132 bash[18607]: cluster 2024-04-25T20:32:33.371720+0000 mgr.x (mgr.14254) 1239 : cluster [DBG] pgmap v1008: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:36.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:36 smithi132 bash[18607]: cluster 2024-04-25T20:32:35.373132+0000 mgr.x (mgr.14254) 1240 : cluster [DBG] pgmap v1009: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:36.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:36 smithi132 bash[18607]: cluster 2024-04-25T20:32:35.373132+0000 mgr.x (mgr.14254) 1240 : cluster [DBG] pgmap v1009: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:38.184 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:32:38.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:38 smithi132 bash[18607]: cluster 2024-04-25T20:32:37.374546+0000 mgr.x (mgr.14254) 1241 : cluster [DBG] pgmap v1010: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:38.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:38 smithi132 bash[18607]: cluster 2024-04-25T20:32:37.374546+0000 mgr.x (mgr.14254) 1241 : cluster [DBG] pgmap v1010: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:40.919 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:32:40.920 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:32:40.932 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:40 smithi132 bash[18607]: cluster 2024-04-25T20:32:39.376405+0000 mgr.x (mgr.14254) 1242 : cluster [DBG] pgmap v1011: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:40.933 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:40 smithi132 bash[18607]: cluster 2024-04-25T20:32:39.376405+0000 mgr.x (mgr.14254) 1242 : cluster [DBG] pgmap v1011: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:41.709 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:32:42.710 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:32:43.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:42 smithi132 bash[18607]: audit 2024-04-25T20:32:40.917860+0000 mgr.x (mgr.14254) 1243 : audit [DBG] from='client.14868 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:32:43.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:42 smithi132 bash[18607]: audit 2024-04-25T20:32:40.917860+0000 mgr.x (mgr.14254) 1243 : audit [DBG] from='client.14868 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:32:43.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:42 smithi132 bash[18607]: cluster 2024-04-25T20:32:41.377889+0000 mgr.x (mgr.14254) 1244 : cluster [DBG] pgmap v1012: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:43.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:42 smithi132 bash[18607]: cluster 2024-04-25T20:32:41.377889+0000 mgr.x (mgr.14254) 1244 : cluster [DBG] pgmap v1012: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:45.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:44 smithi132 bash[18607]: cluster 2024-04-25T20:32:43.379881+0000 mgr.x (mgr.14254) 1245 : cluster [DBG] pgmap v1013: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:45.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:44 smithi132 bash[18607]: cluster 2024-04-25T20:32:43.379881+0000 mgr.x (mgr.14254) 1245 : cluster [DBG] pgmap v1013: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:47.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:46 smithi132 bash[18607]: cluster 2024-04-25T20:32:45.381336+0000 mgr.x (mgr.14254) 1246 : cluster [DBG] pgmap v1014: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:47.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:46 smithi132 bash[18607]: cluster 2024-04-25T20:32:45.381336+0000 mgr.x (mgr.14254) 1246 : cluster [DBG] pgmap v1014: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:47.467 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:32:49.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:48 smithi132 bash[18607]: cluster 2024-04-25T20:32:47.382880+0000 mgr.x (mgr.14254) 1247 : cluster [DBG] pgmap v1015: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:49.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:48 smithi132 bash[18607]: cluster 2024-04-25T20:32:47.382880+0000 mgr.x (mgr.14254) 1247 : cluster [DBG] pgmap v1015: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:50.374 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:32:50.374 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:32:51.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:50 smithi132 bash[18607]: cluster 2024-04-25T20:32:49.384894+0000 mgr.x (mgr.14254) 1248 : cluster [DBG] pgmap v1016: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:51.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:50 smithi132 bash[18607]: cluster 2024-04-25T20:32:49.384894+0000 mgr.x (mgr.14254) 1248 : cluster [DBG] pgmap v1016: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:51.130 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:32:52.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:51 smithi132 bash[18607]: audit 2024-04-25T20:32:50.372382+0000 mgr.x (mgr.14254) 1249 : audit [DBG] from='client.14870 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:32:52.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:51 smithi132 bash[18607]: audit 2024-04-25T20:32:50.372382+0000 mgr.x (mgr.14254) 1249 : audit [DBG] from='client.14870 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:32:52.131 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:32:53.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:52 smithi132 bash[18607]: cluster 2024-04-25T20:32:51.386293+0000 mgr.x (mgr.14254) 1250 : cluster [DBG] pgmap v1017: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:53.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:52 smithi132 bash[18607]: cluster 2024-04-25T20:32:51.386293+0000 mgr.x (mgr.14254) 1250 : cluster [DBG] pgmap v1017: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:55.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:54 smithi132 bash[18607]: cluster 2024-04-25T20:32:53.388142+0000 mgr.x (mgr.14254) 1251 : cluster [DBG] pgmap v1018: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:55.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:54 smithi132 bash[18607]: cluster 2024-04-25T20:32:53.388142+0000 mgr.x (mgr.14254) 1251 : cluster [DBG] pgmap v1018: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:56.889 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:32:57.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:56 smithi132 bash[18607]: cluster 2024-04-25T20:32:55.389570+0000 mgr.x (mgr.14254) 1252 : cluster [DBG] pgmap v1019: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:57.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:56 smithi132 bash[18607]: cluster 2024-04-25T20:32:55.389570+0000 mgr.x (mgr.14254) 1252 : cluster [DBG] pgmap v1019: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:59.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:58 smithi132 bash[18607]: cluster 2024-04-25T20:32:57.390931+0000 mgr.x (mgr.14254) 1253 : cluster [DBG] pgmap v1020: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:59.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:32:58 smithi132 bash[18607]: cluster 2024-04-25T20:32:57.390931+0000 mgr.x (mgr.14254) 1253 : cluster [DBG] pgmap v1020: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:32:59.625 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:32:59.626 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:33:00.301 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:33:01.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:00 smithi132 bash[18607]: cluster 2024-04-25T20:32:59.392593+0000 mgr.x (mgr.14254) 1254 : cluster [DBG] pgmap v1021: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:01.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:00 smithi132 bash[18607]: cluster 2024-04-25T20:32:59.392593+0000 mgr.x (mgr.14254) 1254 : cluster [DBG] pgmap v1021: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:01.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:00 smithi132 bash[18607]: audit 2024-04-25T20:32:59.626508+0000 mgr.x (mgr.14254) 1255 : audit [DBG] from='client.14872 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:33:01.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:00 smithi132 bash[18607]: audit 2024-04-25T20:32:59.626508+0000 mgr.x (mgr.14254) 1255 : audit [DBG] from='client.14872 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:33:01.302 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:33:03.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:02 smithi132 bash[18607]: cluster 2024-04-25T20:33:01.393983+0000 mgr.x (mgr.14254) 1256 : cluster [DBG] pgmap v1022: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:03.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:02 smithi132 bash[18607]: cluster 2024-04-25T20:33:01.393983+0000 mgr.x (mgr.14254) 1256 : cluster [DBG] pgmap v1022: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:04.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:03 smithi132 bash[18607]: audit 2024-04-25T20:33:03.676475+0000 mon.a (mon.0) 522 : audit [DBG] from='client.? 172.17.0.2:0/3811641784' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:33:04.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:03 smithi132 bash[18607]: audit 2024-04-25T20:33:03.676475+0000 mon.a (mon.0) 522 : audit [DBG] from='client.? 172.17.0.2:0/3811641784' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:33:05.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:04 smithi132 bash[18607]: cluster 2024-04-25T20:33:03.395770+0000 mgr.x (mgr.14254) 1257 : cluster [DBG] pgmap v1023: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:05.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:04 smithi132 bash[18607]: cluster 2024-04-25T20:33:03.395770+0000 mgr.x (mgr.14254) 1257 : cluster [DBG] pgmap v1023: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:06.059 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:33:07.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:06 smithi132 bash[18607]: cluster 2024-04-25T20:33:05.397211+0000 mgr.x (mgr.14254) 1258 : cluster [DBG] pgmap v1024: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:07.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:06 smithi132 bash[18607]: cluster 2024-04-25T20:33:05.397211+0000 mgr.x (mgr.14254) 1258 : cluster [DBG] pgmap v1024: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:08.995 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:33:08.995 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:33:09.008 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:08 smithi132 bash[18607]: cluster 2024-04-25T20:33:07.398610+0000 mgr.x (mgr.14254) 1259 : cluster [DBG] pgmap v1025: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:09.008 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:08 smithi132 bash[18607]: cluster 2024-04-25T20:33:07.398610+0000 mgr.x (mgr.14254) 1259 : cluster [DBG] pgmap v1025: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:09.736 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:33:10.737 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:33:11.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:10 smithi132 bash[18607]: audit 2024-04-25T20:33:08.993632+0000 mgr.x (mgr.14254) 1260 : audit [DBG] from='client.14876 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:33:11.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:10 smithi132 bash[18607]: audit 2024-04-25T20:33:08.993632+0000 mgr.x (mgr.14254) 1260 : audit [DBG] from='client.14876 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:33:11.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:10 smithi132 bash[18607]: cluster 2024-04-25T20:33:09.400440+0000 mgr.x (mgr.14254) 1261 : cluster [DBG] pgmap v1026: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:11.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:10 smithi132 bash[18607]: cluster 2024-04-25T20:33:09.400440+0000 mgr.x (mgr.14254) 1261 : cluster [DBG] pgmap v1026: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:13.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:12 smithi132 bash[18607]: cluster 2024-04-25T20:33:11.401860+0000 mgr.x (mgr.14254) 1262 : cluster [DBG] pgmap v1027: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:13.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:12 smithi132 bash[18607]: cluster 2024-04-25T20:33:11.401860+0000 mgr.x (mgr.14254) 1262 : cluster [DBG] pgmap v1027: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:15.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:14 smithi132 bash[18607]: cluster 2024-04-25T20:33:13.403712+0000 mgr.x (mgr.14254) 1263 : cluster [DBG] pgmap v1028: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:15.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:14 smithi132 bash[18607]: cluster 2024-04-25T20:33:13.403712+0000 mgr.x (mgr.14254) 1263 : cluster [DBG] pgmap v1028: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:15.491 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:33:17.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:16 smithi132 bash[18607]: cluster 2024-04-25T20:33:15.405140+0000 mgr.x (mgr.14254) 1264 : cluster [DBG] pgmap v1029: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:17.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:16 smithi132 bash[18607]: cluster 2024-04-25T20:33:15.405140+0000 mgr.x (mgr.14254) 1264 : cluster [DBG] pgmap v1029: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:18.380 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:33:18.380 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:33:19.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:18 smithi132 bash[18607]: cluster 2024-04-25T20:33:17.406458+0000 mgr.x (mgr.14254) 1265 : cluster [DBG] pgmap v1030: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:19.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:18 smithi132 bash[18607]: cluster 2024-04-25T20:33:17.406458+0000 mgr.x (mgr.14254) 1265 : cluster [DBG] pgmap v1030: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:19.086 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:33:20.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:19 smithi132 bash[18607]: audit 2024-04-25T20:33:18.378175+0000 mgr.x (mgr.14254) 1266 : audit [DBG] from='client.14878 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:33:20.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:19 smithi132 bash[18607]: audit 2024-04-25T20:33:18.378175+0000 mgr.x (mgr.14254) 1266 : audit [DBG] from='client.14878 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:33:20.087 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:33:21.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:20 smithi132 bash[18607]: cluster 2024-04-25T20:33:19.408308+0000 mgr.x (mgr.14254) 1267 : cluster [DBG] pgmap v1031: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:21.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:20 smithi132 bash[18607]: cluster 2024-04-25T20:33:19.408308+0000 mgr.x (mgr.14254) 1267 : cluster [DBG] pgmap v1031: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:23.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:22 smithi132 bash[18607]: cluster 2024-04-25T20:33:21.409731+0000 mgr.x (mgr.14254) 1268 : cluster [DBG] pgmap v1032: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:23.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:22 smithi132 bash[18607]: cluster 2024-04-25T20:33:21.409731+0000 mgr.x (mgr.14254) 1268 : cluster [DBG] pgmap v1032: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:24.840 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:33:25.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:24 smithi132 bash[18607]: cluster 2024-04-25T20:33:23.411591+0000 mgr.x (mgr.14254) 1269 : cluster [DBG] pgmap v1033: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:25.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:24 smithi132 bash[18607]: cluster 2024-04-25T20:33:23.411591+0000 mgr.x (mgr.14254) 1269 : cluster [DBG] pgmap v1033: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:26.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:26 smithi132 bash[18607]: cluster 2024-04-25T20:33:25.413136+0000 mgr.x (mgr.14254) 1270 : cluster [DBG] pgmap v1034: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:26.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:26 smithi132 bash[18607]: cluster 2024-04-25T20:33:25.413136+0000 mgr.x (mgr.14254) 1270 : cluster [DBG] pgmap v1034: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:27.583 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:33:27.583 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:33:28.310 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:33:29.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:28 smithi132 bash[18607]: cluster 2024-04-25T20:33:27.414587+0000 mgr.x (mgr.14254) 1271 : cluster [DBG] pgmap v1035: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:29.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:28 smithi132 bash[18607]: cluster 2024-04-25T20:33:27.414587+0000 mgr.x (mgr.14254) 1271 : cluster [DBG] pgmap v1035: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:29.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:28 smithi132 bash[18607]: audit 2024-04-25T20:33:27.581170+0000 mgr.x (mgr.14254) 1272 : audit [DBG] from='client.14880 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:33:29.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:28 smithi132 bash[18607]: audit 2024-04-25T20:33:27.581170+0000 mgr.x (mgr.14254) 1272 : audit [DBG] from='client.14880 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:33:29.311 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:33:31.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:30 smithi132 bash[18607]: cluster 2024-04-25T20:33:29.416378+0000 mgr.x (mgr.14254) 1273 : cluster [DBG] pgmap v1036: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:31.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:30 smithi132 bash[18607]: cluster 2024-04-25T20:33:29.416378+0000 mgr.x (mgr.14254) 1273 : cluster [DBG] pgmap v1036: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:33.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:32 smithi132 bash[18607]: cluster 2024-04-25T20:33:31.417882+0000 mgr.x (mgr.14254) 1274 : cluster [DBG] pgmap v1037: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:33.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:32 smithi132 bash[18607]: cluster 2024-04-25T20:33:31.417882+0000 mgr.x (mgr.14254) 1274 : cluster [DBG] pgmap v1037: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:34.068 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:33:35.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:34 smithi132 bash[18607]: cluster 2024-04-25T20:33:33.419728+0000 mgr.x (mgr.14254) 1275 : cluster [DBG] pgmap v1038: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:35.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:34 smithi132 bash[18607]: cluster 2024-04-25T20:33:33.419728+0000 mgr.x (mgr.14254) 1275 : cluster [DBG] pgmap v1038: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:36.720 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:33:36.720 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:33:37.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:36 smithi132 bash[18607]: cluster 2024-04-25T20:33:35.421154+0000 mgr.x (mgr.14254) 1276 : cluster [DBG] pgmap v1039: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:37.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:36 smithi132 bash[18607]: cluster 2024-04-25T20:33:35.421154+0000 mgr.x (mgr.14254) 1276 : cluster [DBG] pgmap v1039: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:37.415 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:33:38.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:37 smithi132 bash[18607]: audit 2024-04-25T20:33:36.718129+0000 mgr.x (mgr.14254) 1277 : audit [DBG] from='client.14882 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:33:38.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:37 smithi132 bash[18607]: audit 2024-04-25T20:33:36.718129+0000 mgr.x (mgr.14254) 1277 : audit [DBG] from='client.14882 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:33:38.416 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:33:39.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:38 smithi132 bash[18607]: cluster 2024-04-25T20:33:37.422584+0000 mgr.x (mgr.14254) 1278 : cluster [DBG] pgmap v1040: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:39.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:38 smithi132 bash[18607]: cluster 2024-04-25T20:33:37.422584+0000 mgr.x (mgr.14254) 1278 : cluster [DBG] pgmap v1040: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:41.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:40 smithi132 bash[18607]: cluster 2024-04-25T20:33:39.424432+0000 mgr.x (mgr.14254) 1279 : cluster [DBG] pgmap v1041: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:41.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:40 smithi132 bash[18607]: cluster 2024-04-25T20:33:39.424432+0000 mgr.x (mgr.14254) 1279 : cluster [DBG] pgmap v1041: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:43.175 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:33:43.198 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:42 smithi132 bash[18607]: cluster 2024-04-25T20:33:41.425859+0000 mgr.x (mgr.14254) 1280 : cluster [DBG] pgmap v1042: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:43.198 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:42 smithi132 bash[18607]: cluster 2024-04-25T20:33:41.425859+0000 mgr.x (mgr.14254) 1280 : cluster [DBG] pgmap v1042: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:45.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:44 smithi132 bash[18607]: cluster 2024-04-25T20:33:43.426750+0000 mgr.x (mgr.14254) 1281 : cluster [DBG] pgmap v1043: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:45.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:44 smithi132 bash[18607]: cluster 2024-04-25T20:33:43.426750+0000 mgr.x (mgr.14254) 1281 : cluster [DBG] pgmap v1043: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:45.896 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:33:45.896 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:33:46.591 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:33:47.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:46 smithi132 bash[18607]: cluster 2024-04-25T20:33:45.428235+0000 mgr.x (mgr.14254) 1282 : cluster [DBG] pgmap v1044: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:47.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:46 smithi132 bash[18607]: cluster 2024-04-25T20:33:45.428235+0000 mgr.x (mgr.14254) 1282 : cluster [DBG] pgmap v1044: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:47.592 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:33:48.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:47 smithi132 bash[18607]: audit 2024-04-25T20:33:45.896764+0000 mgr.x (mgr.14254) 1283 : audit [DBG] from='client.14884 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:33:48.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:47 smithi132 bash[18607]: audit 2024-04-25T20:33:45.896764+0000 mgr.x (mgr.14254) 1283 : audit [DBG] from='client.14884 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:33:49.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:48 smithi132 bash[18607]: cluster 2024-04-25T20:33:47.429655+0000 mgr.x (mgr.14254) 1284 : cluster [DBG] pgmap v1045: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:49.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:48 smithi132 bash[18607]: cluster 2024-04-25T20:33:47.429655+0000 mgr.x (mgr.14254) 1284 : cluster [DBG] pgmap v1045: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:51.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:50 smithi132 bash[18607]: cluster 2024-04-25T20:33:49.431574+0000 mgr.x (mgr.14254) 1285 : cluster [DBG] pgmap v1046: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:51.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:50 smithi132 bash[18607]: cluster 2024-04-25T20:33:49.431574+0000 mgr.x (mgr.14254) 1285 : cluster [DBG] pgmap v1046: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:52.349 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:33:53.209 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:52 smithi132 bash[18607]: cluster 2024-04-25T20:33:51.433015+0000 mgr.x (mgr.14254) 1286 : cluster [DBG] pgmap v1047: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:53.209 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:52 smithi132 bash[18607]: cluster 2024-04-25T20:33:51.433015+0000 mgr.x (mgr.14254) 1286 : cluster [DBG] pgmap v1047: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:55.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:54 smithi132 bash[18607]: cluster 2024-04-25T20:33:53.434170+0000 mgr.x (mgr.14254) 1287 : cluster [DBG] pgmap v1048: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:55.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:54 smithi132 bash[18607]: cluster 2024-04-25T20:33:53.434170+0000 mgr.x (mgr.14254) 1287 : cluster [DBG] pgmap v1048: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:55.125 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:33:55.125 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:33:55.925 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:33:56.925 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:33:57.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:56 smithi132 bash[18607]: audit 2024-04-25T20:33:55.126341+0000 mgr.x (mgr.14254) 1288 : audit [DBG] from='client.14886 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:33:57.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:56 smithi132 bash[18607]: audit 2024-04-25T20:33:55.126341+0000 mgr.x (mgr.14254) 1288 : audit [DBG] from='client.14886 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:33:57.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:56 smithi132 bash[18607]: cluster 2024-04-25T20:33:55.435524+0000 mgr.x (mgr.14254) 1289 : cluster [DBG] pgmap v1049: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:57.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:56 smithi132 bash[18607]: cluster 2024-04-25T20:33:55.435524+0000 mgr.x (mgr.14254) 1289 : cluster [DBG] pgmap v1049: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:59.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:58 smithi132 bash[18607]: cluster 2024-04-25T20:33:57.436937+0000 mgr.x (mgr.14254) 1290 : cluster [DBG] pgmap v1050: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:33:59.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:33:58 smithi132 bash[18607]: cluster 2024-04-25T20:33:57.436937+0000 mgr.x (mgr.14254) 1290 : cluster [DBG] pgmap v1050: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:01.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:00 smithi132 bash[18607]: cluster 2024-04-25T20:33:59.438725+0000 mgr.x (mgr.14254) 1291 : cluster [DBG] pgmap v1051: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:01.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:00 smithi132 bash[18607]: cluster 2024-04-25T20:33:59.438725+0000 mgr.x (mgr.14254) 1291 : cluster [DBG] pgmap v1051: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:01.676 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:34:03.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:02 smithi132 bash[18607]: cluster 2024-04-25T20:34:01.440105+0000 mgr.x (mgr.14254) 1292 : cluster [DBG] pgmap v1052: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:03.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:02 smithi132 bash[18607]: cluster 2024-04-25T20:34:01.440105+0000 mgr.x (mgr.14254) 1292 : cluster [DBG] pgmap v1052: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:04.357 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:34:04.357 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:34:05.039 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:34:05.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:04 smithi132 bash[18607]: cluster 2024-04-25T20:34:03.442025+0000 mgr.x (mgr.14254) 1293 : cluster [DBG] pgmap v1053: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:05.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:04 smithi132 bash[18607]: cluster 2024-04-25T20:34:03.442025+0000 mgr.x (mgr.14254) 1293 : cluster [DBG] pgmap v1053: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:06.039 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:34:06.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:05 smithi132 bash[18607]: audit 2024-04-25T20:34:04.355262+0000 mgr.x (mgr.14254) 1294 : audit [DBG] from='client.14888 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:34:06.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:05 smithi132 bash[18607]: audit 2024-04-25T20:34:04.355262+0000 mgr.x (mgr.14254) 1294 : audit [DBG] from='client.14888 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:34:07.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:06 smithi132 bash[18607]: cluster 2024-04-25T20:34:05.443395+0000 mgr.x (mgr.14254) 1295 : cluster [DBG] pgmap v1054: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:07.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:06 smithi132 bash[18607]: cluster 2024-04-25T20:34:05.443395+0000 mgr.x (mgr.14254) 1295 : cluster [DBG] pgmap v1054: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:08.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:07 smithi132 bash[18607]: cluster 2024-04-25T20:34:07.444736+0000 mgr.x (mgr.14254) 1296 : cluster [DBG] pgmap v1055: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:08.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:07 smithi132 bash[18607]: cluster 2024-04-25T20:34:07.444736+0000 mgr.x (mgr.14254) 1296 : cluster [DBG] pgmap v1055: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:10.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:10 smithi132 bash[18607]: cluster 2024-04-25T20:34:09.446486+0000 mgr.x (mgr.14254) 1297 : cluster [DBG] pgmap v1056: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:10.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:10 smithi132 bash[18607]: cluster 2024-04-25T20:34:09.446486+0000 mgr.x (mgr.14254) 1297 : cluster [DBG] pgmap v1056: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:10.796 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:34:12.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:12 smithi132 bash[18607]: cluster 2024-04-25T20:34:11.447473+0000 mgr.x (mgr.14254) 1298 : cluster [DBG] pgmap v1057: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:12.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:12 smithi132 bash[18607]: cluster 2024-04-25T20:34:11.447473+0000 mgr.x (mgr.14254) 1298 : cluster [DBG] pgmap v1057: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:13.704 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:34:13.705 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:34:14.363 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:34:14.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:14 smithi132 bash[18607]: cluster 2024-04-25T20:34:13.449283+0000 mgr.x (mgr.14254) 1299 : cluster [DBG] pgmap v1058: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:14.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:14 smithi132 bash[18607]: cluster 2024-04-25T20:34:13.449283+0000 mgr.x (mgr.14254) 1299 : cluster [DBG] pgmap v1058: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:14.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:14 smithi132 bash[18607]: audit 2024-04-25T20:34:13.702949+0000 mgr.x (mgr.14254) 1300 : audit [DBG] from='client.14890 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:34:14.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:14 smithi132 bash[18607]: audit 2024-04-25T20:34:13.702949+0000 mgr.x (mgr.14254) 1300 : audit [DBG] from='client.14890 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:34:14.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:14 smithi132 bash[18607]: audit 2024-04-25T20:34:14.068886+0000 mon.a (mon.0) 523 : audit [DBG] from='client.? 172.17.0.2:0/1552531816' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:34:14.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:14 smithi132 bash[18607]: audit 2024-04-25T20:34:14.068886+0000 mon.a (mon.0) 523 : audit [DBG] from='client.? 172.17.0.2:0/1552531816' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:34:15.364 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:34:16.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:16 smithi132 bash[18607]: cluster 2024-04-25T20:34:15.450672+0000 mgr.x (mgr.14254) 1301 : cluster [DBG] pgmap v1059: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:16.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:16 smithi132 bash[18607]: cluster 2024-04-25T20:34:15.450672+0000 mgr.x (mgr.14254) 1301 : cluster [DBG] pgmap v1059: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:18.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:18 smithi132 bash[18607]: cluster 2024-04-25T20:34:17.452232+0000 mgr.x (mgr.14254) 1302 : cluster [DBG] pgmap v1060: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:18.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:18 smithi132 bash[18607]: cluster 2024-04-25T20:34:17.452232+0000 mgr.x (mgr.14254) 1302 : cluster [DBG] pgmap v1060: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:20.121 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:34:20.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:20 smithi132 bash[18607]: cluster 2024-04-25T20:34:19.454170+0000 mgr.x (mgr.14254) 1303 : cluster [DBG] pgmap v1061: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:20.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:20 smithi132 bash[18607]: cluster 2024-04-25T20:34:19.454170+0000 mgr.x (mgr.14254) 1303 : cluster [DBG] pgmap v1061: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:22.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:22 smithi132 bash[18607]: cluster 2024-04-25T20:34:21.455542+0000 mgr.x (mgr.14254) 1304 : cluster [DBG] pgmap v1062: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:22.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:22 smithi132 bash[18607]: cluster 2024-04-25T20:34:21.455542+0000 mgr.x (mgr.14254) 1304 : cluster [DBG] pgmap v1062: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:22.906 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:34:22.906 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:34:23.634 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:34:24.635 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:34:24.646 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:24 smithi132 bash[18607]: audit 2024-04-25T20:34:22.907992+0000 mgr.x (mgr.14254) 1305 : audit [DBG] from='client.14894 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:34:24.646 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:24 smithi132 bash[18607]: audit 2024-04-25T20:34:22.907992+0000 mgr.x (mgr.14254) 1305 : audit [DBG] from='client.14894 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:34:24.646 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:24 smithi132 bash[18607]: cluster 2024-04-25T20:34:23.457382+0000 mgr.x (mgr.14254) 1306 : cluster [DBG] pgmap v1063: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:24.646 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:24 smithi132 bash[18607]: cluster 2024-04-25T20:34:23.457382+0000 mgr.x (mgr.14254) 1306 : cluster [DBG] pgmap v1063: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:26.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:26 smithi132 bash[18607]: cluster 2024-04-25T20:34:25.458760+0000 mgr.x (mgr.14254) 1307 : cluster [DBG] pgmap v1064: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:26.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:26 smithi132 bash[18607]: cluster 2024-04-25T20:34:25.458760+0000 mgr.x (mgr.14254) 1307 : cluster [DBG] pgmap v1064: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:28.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:28 smithi132 bash[18607]: cluster 2024-04-25T20:34:27.460230+0000 mgr.x (mgr.14254) 1308 : cluster [DBG] pgmap v1065: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:28.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:28 smithi132 bash[18607]: cluster 2024-04-25T20:34:27.460230+0000 mgr.x (mgr.14254) 1308 : cluster [DBG] pgmap v1065: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:29.395 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:34:30.579 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:30 smithi132 bash[18607]: cluster 2024-04-25T20:34:29.461221+0000 mgr.x (mgr.14254) 1309 : cluster [DBG] pgmap v1066: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:30.579 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:30 smithi132 bash[18607]: cluster 2024-04-25T20:34:29.461221+0000 mgr.x (mgr.14254) 1309 : cluster [DBG] pgmap v1066: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:32.134 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:34:32.134 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:34:32.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:32 smithi132 bash[18607]: cluster 2024-04-25T20:34:31.462579+0000 mgr.x (mgr.14254) 1310 : cluster [DBG] pgmap v1067: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:32.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:32 smithi132 bash[18607]: cluster 2024-04-25T20:34:31.462579+0000 mgr.x (mgr.14254) 1310 : cluster [DBG] pgmap v1067: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:32.795 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:34:33.796 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:34:33.807 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:33 smithi132 bash[18607]: audit 2024-04-25T20:34:32.132526+0000 mgr.x (mgr.14254) 1311 : audit [DBG] from='client.14896 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:34:33.808 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:33 smithi132 bash[18607]: audit 2024-04-25T20:34:32.132526+0000 mgr.x (mgr.14254) 1311 : audit [DBG] from='client.14896 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:34:35.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:34 smithi132 bash[18607]: cluster 2024-04-25T20:34:33.464389+0000 mgr.x (mgr.14254) 1312 : cluster [DBG] pgmap v1068: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:35.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:34 smithi132 bash[18607]: cluster 2024-04-25T20:34:33.464389+0000 mgr.x (mgr.14254) 1312 : cluster [DBG] pgmap v1068: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:37.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:36 smithi132 bash[18607]: cluster 2024-04-25T20:34:35.465770+0000 mgr.x (mgr.14254) 1313 : cluster [DBG] pgmap v1069: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:37.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:36 smithi132 bash[18607]: cluster 2024-04-25T20:34:35.465770+0000 mgr.x (mgr.14254) 1313 : cluster [DBG] pgmap v1069: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:38.561 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:34:39.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:38 smithi132 bash[18607]: cluster 2024-04-25T20:34:37.467113+0000 mgr.x (mgr.14254) 1314 : cluster [DBG] pgmap v1070: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:39.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:38 smithi132 bash[18607]: cluster 2024-04-25T20:34:37.467113+0000 mgr.x (mgr.14254) 1314 : cluster [DBG] pgmap v1070: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:40.944 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:40 smithi132 bash[18607]: cluster 2024-04-25T20:34:39.467979+0000 mgr.x (mgr.14254) 1315 : cluster [DBG] pgmap v1071: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:40.945 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:40 smithi132 bash[18607]: cluster 2024-04-25T20:34:39.467979+0000 mgr.x (mgr.14254) 1315 : cluster [DBG] pgmap v1071: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:41.357 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:34:41.357 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:34:42.067 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:34:43.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:42 smithi132 bash[18607]: audit 2024-04-25T20:34:41.357397+0000 mgr.x (mgr.14254) 1316 : audit [DBG] from='client.14898 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:34:43.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:42 smithi132 bash[18607]: audit 2024-04-25T20:34:41.357397+0000 mgr.x (mgr.14254) 1316 : audit [DBG] from='client.14898 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:34:43.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:42 smithi132 bash[18607]: cluster 2024-04-25T20:34:41.469380+0000 mgr.x (mgr.14254) 1317 : cluster [DBG] pgmap v1072: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:43.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:42 smithi132 bash[18607]: cluster 2024-04-25T20:34:41.469380+0000 mgr.x (mgr.14254) 1317 : cluster [DBG] pgmap v1072: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:43.069 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:34:45.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:44 smithi132 bash[18607]: cluster 2024-04-25T20:34:43.471237+0000 mgr.x (mgr.14254) 1318 : cluster [DBG] pgmap v1073: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:45.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:44 smithi132 bash[18607]: cluster 2024-04-25T20:34:43.471237+0000 mgr.x (mgr.14254) 1318 : cluster [DBG] pgmap v1073: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:47.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:46 smithi132 bash[18607]: cluster 2024-04-25T20:34:45.472597+0000 mgr.x (mgr.14254) 1319 : cluster [DBG] pgmap v1074: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:47.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:46 smithi132 bash[18607]: cluster 2024-04-25T20:34:45.472597+0000 mgr.x (mgr.14254) 1319 : cluster [DBG] pgmap v1074: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:47.832 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:34:48.901 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:48 smithi132 bash[18607]: cluster 2024-04-25T20:34:47.474004+0000 mgr.x (mgr.14254) 1320 : cluster [DBG] pgmap v1075: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:48.901 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:48 smithi132 bash[18607]: cluster 2024-04-25T20:34:47.474004+0000 mgr.x (mgr.14254) 1320 : cluster [DBG] pgmap v1075: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:50.582 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:34:50.582 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:34:50.792 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:50 smithi132 bash[18607]: cluster 2024-04-25T20:34:49.474947+0000 mgr.x (mgr.14254) 1321 : cluster [DBG] pgmap v1076: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:50.792 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:50 smithi132 bash[18607]: cluster 2024-04-25T20:34:49.474947+0000 mgr.x (mgr.14254) 1321 : cluster [DBG] pgmap v1076: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:51.319 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:34:51.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:51 smithi132 bash[18607]: audit 2024-04-25T20:34:50.582722+0000 mgr.x (mgr.14254) 1322 : audit [DBG] from='client.14900 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:34:51.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:51 smithi132 bash[18607]: audit 2024-04-25T20:34:50.582722+0000 mgr.x (mgr.14254) 1322 : audit [DBG] from='client.14900 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:34:52.320 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:34:52.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:52 smithi132 bash[18607]: cluster 2024-04-25T20:34:51.476349+0000 mgr.x (mgr.14254) 1323 : cluster [DBG] pgmap v1077: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:52.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:52 smithi132 bash[18607]: cluster 2024-04-25T20:34:51.476349+0000 mgr.x (mgr.14254) 1323 : cluster [DBG] pgmap v1077: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:55.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:54 smithi132 bash[18607]: cluster 2024-04-25T20:34:53.478250+0000 mgr.x (mgr.14254) 1324 : cluster [DBG] pgmap v1078: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:55.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:54 smithi132 bash[18607]: cluster 2024-04-25T20:34:53.478250+0000 mgr.x (mgr.14254) 1324 : cluster [DBG] pgmap v1078: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:57.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:56 smithi132 bash[18607]: cluster 2024-04-25T20:34:55.479622+0000 mgr.x (mgr.14254) 1325 : cluster [DBG] pgmap v1079: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:57.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:56 smithi132 bash[18607]: cluster 2024-04-25T20:34:55.479622+0000 mgr.x (mgr.14254) 1325 : cluster [DBG] pgmap v1079: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:57.079 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:34:58.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:58 smithi132 bash[18607]: cluster 2024-04-25T20:34:57.481041+0000 mgr.x (mgr.14254) 1326 : cluster [DBG] pgmap v1080: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:58.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:34:58 smithi132 bash[18607]: cluster 2024-04-25T20:34:57.481041+0000 mgr.x (mgr.14254) 1326 : cluster [DBG] pgmap v1080: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:34:59.968 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:34:59.968 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:35:00.690 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:35:01.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:00 smithi132 bash[18607]: cluster 2024-04-25T20:34:59.482912+0000 mgr.x (mgr.14254) 1327 : cluster [DBG] pgmap v1081: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:01.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:00 smithi132 bash[18607]: cluster 2024-04-25T20:34:59.482912+0000 mgr.x (mgr.14254) 1327 : cluster [DBG] pgmap v1081: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:01.691 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:35:02.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:01 smithi132 bash[18607]: audit 2024-04-25T20:34:59.967659+0000 mgr.x (mgr.14254) 1328 : audit [DBG] from='client.14902 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:35:02.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:01 smithi132 bash[18607]: audit 2024-04-25T20:34:59.967659+0000 mgr.x (mgr.14254) 1328 : audit [DBG] from='client.14902 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:35:03.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:02 smithi132 bash[18607]: cluster 2024-04-25T20:35:01.484386+0000 mgr.x (mgr.14254) 1329 : cluster [DBG] pgmap v1082: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:03.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:02 smithi132 bash[18607]: cluster 2024-04-25T20:35:01.484386+0000 mgr.x (mgr.14254) 1329 : cluster [DBG] pgmap v1082: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:05.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:04 smithi132 bash[18607]: cluster 2024-04-25T20:35:03.486209+0000 mgr.x (mgr.14254) 1330 : cluster [DBG] pgmap v1083: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:05.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:04 smithi132 bash[18607]: cluster 2024-04-25T20:35:03.486209+0000 mgr.x (mgr.14254) 1330 : cluster [DBG] pgmap v1083: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:06.448 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:35:06.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:06 smithi132 bash[18607]: cluster 2024-04-25T20:35:05.487635+0000 mgr.x (mgr.14254) 1331 : cluster [DBG] pgmap v1084: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:06.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:06 smithi132 bash[18607]: cluster 2024-04-25T20:35:05.487635+0000 mgr.x (mgr.14254) 1331 : cluster [DBG] pgmap v1084: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:08.827 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:08 smithi132 bash[18607]: cluster 2024-04-25T20:35:07.488454+0000 mgr.x (mgr.14254) 1332 : cluster [DBG] pgmap v1085: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:08.827 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:08 smithi132 bash[18607]: cluster 2024-04-25T20:35:07.488454+0000 mgr.x (mgr.14254) 1332 : cluster [DBG] pgmap v1085: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:09.204 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:35:09.204 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:35:09.964 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:35:10.965 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:35:10.976 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:10 smithi132 bash[18607]: audit 2024-04-25T20:35:09.202334+0000 mgr.x (mgr.14254) 1333 : audit [DBG] from='client.14904 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:35:10.976 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:10 smithi132 bash[18607]: audit 2024-04-25T20:35:09.202334+0000 mgr.x (mgr.14254) 1333 : audit [DBG] from='client.14904 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:35:10.976 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:10 smithi132 bash[18607]: cluster 2024-04-25T20:35:09.490417+0000 mgr.x (mgr.14254) 1334 : cluster [DBG] pgmap v1086: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:10.976 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:10 smithi132 bash[18607]: cluster 2024-04-25T20:35:09.490417+0000 mgr.x (mgr.14254) 1334 : cluster [DBG] pgmap v1086: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:13.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:12 smithi132 bash[18607]: cluster 2024-04-25T20:35:11.491966+0000 mgr.x (mgr.14254) 1335 : cluster [DBG] pgmap v1087: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:13.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:12 smithi132 bash[18607]: cluster 2024-04-25T20:35:11.491966+0000 mgr.x (mgr.14254) 1335 : cluster [DBG] pgmap v1087: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:15.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:14 smithi132 bash[18607]: cluster 2024-04-25T20:35:13.493918+0000 mgr.x (mgr.14254) 1336 : cluster [DBG] pgmap v1088: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:15.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:14 smithi132 bash[18607]: cluster 2024-04-25T20:35:13.493918+0000 mgr.x (mgr.14254) 1336 : cluster [DBG] pgmap v1088: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:15.721 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:35:16.872 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:16 smithi132 bash[18607]: cluster 2024-04-25T20:35:15.495339+0000 mgr.x (mgr.14254) 1337 : cluster [DBG] pgmap v1089: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:16.872 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:16 smithi132 bash[18607]: cluster 2024-04-25T20:35:15.495339+0000 mgr.x (mgr.14254) 1337 : cluster [DBG] pgmap v1089: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:18.580 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:35:18.580 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:35:19.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:18 smithi132 bash[18607]: cluster 2024-04-25T20:35:17.496746+0000 mgr.x (mgr.14254) 1338 : cluster [DBG] pgmap v1090: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:19.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:18 smithi132 bash[18607]: cluster 2024-04-25T20:35:17.496746+0000 mgr.x (mgr.14254) 1338 : cluster [DBG] pgmap v1090: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:19.246 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:35:20.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:19 smithi132 bash[18607]: audit 2024-04-25T20:35:18.581519+0000 mgr.x (mgr.14254) 1339 : audit [DBG] from='client.14906 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:35:20.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:19 smithi132 bash[18607]: audit 2024-04-25T20:35:18.581519+0000 mgr.x (mgr.14254) 1339 : audit [DBG] from='client.14906 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:35:20.246 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:35:21.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:20 smithi132 bash[18607]: cluster 2024-04-25T20:35:19.498524+0000 mgr.x (mgr.14254) 1340 : cluster [DBG] pgmap v1091: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:21.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:20 smithi132 bash[18607]: cluster 2024-04-25T20:35:19.498524+0000 mgr.x (mgr.14254) 1340 : cluster [DBG] pgmap v1091: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:23.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:22 smithi132 bash[18607]: cluster 2024-04-25T20:35:21.499894+0000 mgr.x (mgr.14254) 1341 : cluster [DBG] pgmap v1092: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:23.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:22 smithi132 bash[18607]: cluster 2024-04-25T20:35:21.499894+0000 mgr.x (mgr.14254) 1341 : cluster [DBG] pgmap v1092: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:24.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:24 smithi132 bash[18607]: cluster 2024-04-25T20:35:23.501797+0000 mgr.x (mgr.14254) 1342 : cluster [DBG] pgmap v1093: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:24.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:24 smithi132 bash[18607]: cluster 2024-04-25T20:35:23.501797+0000 mgr.x (mgr.14254) 1342 : cluster [DBG] pgmap v1093: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:24.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:24 smithi132 bash[18607]: audit 2024-04-25T20:35:24.448300+0000 mon.a (mon.0) 524 : audit [DBG] from='client.? 172.17.0.2:0/2561152265' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:35:24.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:24 smithi132 bash[18607]: audit 2024-04-25T20:35:24.448300+0000 mon.a (mon.0) 524 : audit [DBG] from='client.? 172.17.0.2:0/2561152265' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:35:25.005 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:35:27.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:26 smithi132 bash[18607]: cluster 2024-04-25T20:35:25.503223+0000 mgr.x (mgr.14254) 1343 : cluster [DBG] pgmap v1094: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:27.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:26 smithi132 bash[18607]: cluster 2024-04-25T20:35:25.503223+0000 mgr.x (mgr.14254) 1343 : cluster [DBG] pgmap v1094: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:27.931 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:35:27.931 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:35:28.597 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:35:29.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:28 smithi132 bash[18607]: cluster 2024-04-25T20:35:27.504633+0000 mgr.x (mgr.14254) 1344 : cluster [DBG] pgmap v1095: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:29.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:28 smithi132 bash[18607]: cluster 2024-04-25T20:35:27.504633+0000 mgr.x (mgr.14254) 1344 : cluster [DBG] pgmap v1095: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:29.598 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:35:30.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:29 smithi132 bash[18607]: audit 2024-04-25T20:35:27.932270+0000 mgr.x (mgr.14254) 1345 : audit [DBG] from='client.14910 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:35:30.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:29 smithi132 bash[18607]: audit 2024-04-25T20:35:27.932270+0000 mgr.x (mgr.14254) 1345 : audit [DBG] from='client.14910 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:35:31.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:30 smithi132 bash[18607]: cluster 2024-04-25T20:35:29.506478+0000 mgr.x (mgr.14254) 1346 : cluster [DBG] pgmap v1096: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:31.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:30 smithi132 bash[18607]: cluster 2024-04-25T20:35:29.506478+0000 mgr.x (mgr.14254) 1346 : cluster [DBG] pgmap v1096: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:33.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:32 smithi132 bash[18607]: cluster 2024-04-25T20:35:31.507877+0000 mgr.x (mgr.14254) 1347 : cluster [DBG] pgmap v1097: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:33.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:32 smithi132 bash[18607]: cluster 2024-04-25T20:35:31.507877+0000 mgr.x (mgr.14254) 1347 : cluster [DBG] pgmap v1097: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:34.356 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:35:34.752 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:34 smithi132 bash[18607]: cluster 2024-04-25T20:35:33.509717+0000 mgr.x (mgr.14254) 1348 : cluster [DBG] pgmap v1098: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:34.752 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:34 smithi132 bash[18607]: cluster 2024-04-25T20:35:33.509717+0000 mgr.x (mgr.14254) 1348 : cluster [DBG] pgmap v1098: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:36.985 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:36 smithi132 bash[18607]: cluster 2024-04-25T20:35:35.510505+0000 mgr.x (mgr.14254) 1349 : cluster [DBG] pgmap v1099: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:36.985 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:36 smithi132 bash[18607]: cluster 2024-04-25T20:35:35.510505+0000 mgr.x (mgr.14254) 1349 : cluster [DBG] pgmap v1099: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:37.414 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:35:37.414 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:35:38.143 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:35:39.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:38 smithi132 bash[18607]: audit 2024-04-25T20:35:37.411972+0000 mgr.x (mgr.14254) 1350 : audit [DBG] from='client.14912 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:35:39.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:38 smithi132 bash[18607]: audit 2024-04-25T20:35:37.411972+0000 mgr.x (mgr.14254) 1350 : audit [DBG] from='client.14912 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:35:39.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:38 smithi132 bash[18607]: cluster 2024-04-25T20:35:37.511401+0000 mgr.x (mgr.14254) 1351 : cluster [DBG] pgmap v1100: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:39.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:38 smithi132 bash[18607]: cluster 2024-04-25T20:35:37.511401+0000 mgr.x (mgr.14254) 1351 : cluster [DBG] pgmap v1100: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:39.144 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:35:41.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:40 smithi132 bash[18607]: cluster 2024-04-25T20:35:39.513273+0000 mgr.x (mgr.14254) 1352 : cluster [DBG] pgmap v1101: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:41.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:40 smithi132 bash[18607]: cluster 2024-04-25T20:35:39.513273+0000 mgr.x (mgr.14254) 1352 : cluster [DBG] pgmap v1101: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:43.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:42 smithi132 bash[18607]: cluster 2024-04-25T20:35:41.514719+0000 mgr.x (mgr.14254) 1353 : cluster [DBG] pgmap v1102: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:43.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:42 smithi132 bash[18607]: cluster 2024-04-25T20:35:41.514719+0000 mgr.x (mgr.14254) 1353 : cluster [DBG] pgmap v1102: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:43.901 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:35:44.865 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:44 smithi132 bash[18607]: cluster 2024-04-25T20:35:43.516805+0000 mgr.x (mgr.14254) 1354 : cluster [DBG] pgmap v1103: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:44.866 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:44 smithi132 bash[18607]: cluster 2024-04-25T20:35:43.516805+0000 mgr.x (mgr.14254) 1354 : cluster [DBG] pgmap v1103: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:46.614 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:35:46.615 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:35:47.060 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:46 smithi132 bash[18607]: cluster 2024-04-25T20:35:45.517797+0000 mgr.x (mgr.14254) 1355 : cluster [DBG] pgmap v1104: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:47.060 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:46 smithi132 bash[18607]: cluster 2024-04-25T20:35:45.517797+0000 mgr.x (mgr.14254) 1355 : cluster [DBG] pgmap v1104: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:47.364 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:35:48.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:47 smithi132 bash[18607]: audit 2024-04-25T20:35:46.612941+0000 mgr.x (mgr.14254) 1356 : audit [DBG] from='client.14914 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:35:48.022 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:47 smithi132 bash[18607]: audit 2024-04-25T20:35:46.612941+0000 mgr.x (mgr.14254) 1356 : audit [DBG] from='client.14914 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:35:48.365 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:35:48.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:48 smithi132 bash[18607]: cluster 2024-04-25T20:35:47.519218+0000 mgr.x (mgr.14254) 1357 : cluster [DBG] pgmap v1105: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:48.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:48 smithi132 bash[18607]: cluster 2024-04-25T20:35:47.519218+0000 mgr.x (mgr.14254) 1357 : cluster [DBG] pgmap v1105: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:51.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:50 smithi132 bash[18607]: cluster 2024-04-25T20:35:49.521085+0000 mgr.x (mgr.14254) 1358 : cluster [DBG] pgmap v1106: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:51.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:50 smithi132 bash[18607]: cluster 2024-04-25T20:35:49.521085+0000 mgr.x (mgr.14254) 1358 : cluster [DBG] pgmap v1106: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:53.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:52 smithi132 bash[18607]: cluster 2024-04-25T20:35:51.522433+0000 mgr.x (mgr.14254) 1359 : cluster [DBG] pgmap v1107: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:53.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:52 smithi132 bash[18607]: cluster 2024-04-25T20:35:51.522433+0000 mgr.x (mgr.14254) 1359 : cluster [DBG] pgmap v1107: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:53.126 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:35:55.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:54 smithi132 bash[18607]: cluster 2024-04-25T20:35:53.524281+0000 mgr.x (mgr.14254) 1360 : cluster [DBG] pgmap v1108: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:55.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:54 smithi132 bash[18607]: cluster 2024-04-25T20:35:53.524281+0000 mgr.x (mgr.14254) 1360 : cluster [DBG] pgmap v1108: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:56.004 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:35:56.004 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:35:56.667 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:35:57.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:56 smithi132 bash[18607]: cluster 2024-04-25T20:35:55.525679+0000 mgr.x (mgr.14254) 1361 : cluster [DBG] pgmap v1109: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:57.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:56 smithi132 bash[18607]: cluster 2024-04-25T20:35:55.525679+0000 mgr.x (mgr.14254) 1361 : cluster [DBG] pgmap v1109: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:57.668 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:35:58.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:57 smithi132 bash[18607]: audit 2024-04-25T20:35:56.002212+0000 mgr.x (mgr.14254) 1362 : audit [DBG] from='client.14916 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:35:58.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:57 smithi132 bash[18607]: audit 2024-04-25T20:35:56.002212+0000 mgr.x (mgr.14254) 1362 : audit [DBG] from='client.14916 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:35:59.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:58 smithi132 bash[18607]: cluster 2024-04-25T20:35:57.527211+0000 mgr.x (mgr.14254) 1363 : cluster [DBG] pgmap v1110: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:35:59.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:35:58 smithi132 bash[18607]: cluster 2024-04-25T20:35:57.527211+0000 mgr.x (mgr.14254) 1363 : cluster [DBG] pgmap v1110: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:01.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:00 smithi132 bash[18607]: cluster 2024-04-25T20:35:59.529152+0000 mgr.x (mgr.14254) 1364 : cluster [DBG] pgmap v1111: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:01.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:00 smithi132 bash[18607]: cluster 2024-04-25T20:35:59.529152+0000 mgr.x (mgr.14254) 1364 : cluster [DBG] pgmap v1111: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:02.431 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:36:03.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:02 smithi132 bash[18607]: cluster 2024-04-25T20:36:01.530582+0000 mgr.x (mgr.14254) 1365 : cluster [DBG] pgmap v1112: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:03.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:02 smithi132 bash[18607]: cluster 2024-04-25T20:36:01.530582+0000 mgr.x (mgr.14254) 1365 : cluster [DBG] pgmap v1112: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:05.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:04 smithi132 bash[18607]: cluster 2024-04-25T20:36:03.532492+0000 mgr.x (mgr.14254) 1366 : cluster [DBG] pgmap v1113: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:05.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:04 smithi132 bash[18607]: cluster 2024-04-25T20:36:03.532492+0000 mgr.x (mgr.14254) 1366 : cluster [DBG] pgmap v1113: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:05.061 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:36:05.061 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:36:05.781 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:36:06.782 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:36:07.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:06 smithi132 bash[18607]: audit 2024-04-25T20:36:05.063168+0000 mgr.x (mgr.14254) 1367 : audit [DBG] from='client.14918 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:36:07.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:06 smithi132 bash[18607]: audit 2024-04-25T20:36:05.063168+0000 mgr.x (mgr.14254) 1367 : audit [DBG] from='client.14918 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:36:07.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:06 smithi132 bash[18607]: cluster 2024-04-25T20:36:05.533909+0000 mgr.x (mgr.14254) 1368 : cluster [DBG] pgmap v1114: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:07.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:06 smithi132 bash[18607]: cluster 2024-04-25T20:36:05.533909+0000 mgr.x (mgr.14254) 1368 : cluster [DBG] pgmap v1114: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:09.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:08 smithi132 bash[18607]: cluster 2024-04-25T20:36:07.535420+0000 mgr.x (mgr.14254) 1369 : cluster [DBG] pgmap v1115: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:09.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:08 smithi132 bash[18607]: cluster 2024-04-25T20:36:07.535420+0000 mgr.x (mgr.14254) 1369 : cluster [DBG] pgmap v1115: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:11.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:10 smithi132 bash[18607]: cluster 2024-04-25T20:36:09.536588+0000 mgr.x (mgr.14254) 1370 : cluster [DBG] pgmap v1116: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:11.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:10 smithi132 bash[18607]: cluster 2024-04-25T20:36:09.536588+0000 mgr.x (mgr.14254) 1370 : cluster [DBG] pgmap v1116: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:11.540 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:36:13.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:12 smithi132 bash[18607]: cluster 2024-04-25T20:36:11.538046+0000 mgr.x (mgr.14254) 1371 : cluster [DBG] pgmap v1117: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:13.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:12 smithi132 bash[18607]: cluster 2024-04-25T20:36:11.538046+0000 mgr.x (mgr.14254) 1371 : cluster [DBG] pgmap v1117: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:14.365 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:36:14.365 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:36:15.010 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:36:15.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:14 smithi132 bash[18607]: cluster 2024-04-25T20:36:13.540497+0000 mgr.x (mgr.14254) 1372 : cluster [DBG] pgmap v1118: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:15.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:14 smithi132 bash[18607]: cluster 2024-04-25T20:36:13.540497+0000 mgr.x (mgr.14254) 1372 : cluster [DBG] pgmap v1118: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:16.011 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:36:16.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:15 smithi132 bash[18607]: audit 2024-04-25T20:36:14.363251+0000 mgr.x (mgr.14254) 1373 : audit [DBG] from='client.14920 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:36:16.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:15 smithi132 bash[18607]: audit 2024-04-25T20:36:14.363251+0000 mgr.x (mgr.14254) 1373 : audit [DBG] from='client.14920 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:36:17.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:16 smithi132 bash[18607]: cluster 2024-04-25T20:36:15.541912+0000 mgr.x (mgr.14254) 1374 : cluster [DBG] pgmap v1119: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:17.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:16 smithi132 bash[18607]: cluster 2024-04-25T20:36:15.541912+0000 mgr.x (mgr.14254) 1374 : cluster [DBG] pgmap v1119: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:19.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:18 smithi132 bash[18607]: cluster 2024-04-25T20:36:17.543292+0000 mgr.x (mgr.14254) 1375 : cluster [DBG] pgmap v1120: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:19.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:18 smithi132 bash[18607]: cluster 2024-04-25T20:36:17.543292+0000 mgr.x (mgr.14254) 1375 : cluster [DBG] pgmap v1120: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:20.765 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:36:21.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:20 smithi132 bash[18607]: cluster 2024-04-25T20:36:19.545158+0000 mgr.x (mgr.14254) 1376 : cluster [DBG] pgmap v1121: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:21.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:20 smithi132 bash[18607]: cluster 2024-04-25T20:36:19.545158+0000 mgr.x (mgr.14254) 1376 : cluster [DBG] pgmap v1121: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:23.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:22 smithi132 bash[18607]: cluster 2024-04-25T20:36:21.546560+0000 mgr.x (mgr.14254) 1377 : cluster [DBG] pgmap v1122: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:23.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:22 smithi132 bash[18607]: cluster 2024-04-25T20:36:21.546560+0000 mgr.x (mgr.14254) 1377 : cluster [DBG] pgmap v1122: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:23.433 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:36:23.433 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:36:24.142 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:36:25.143 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:36:25.154 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:24 smithi132 bash[18607]: audit 2024-04-25T20:36:23.431047+0000 mgr.x (mgr.14254) 1378 : audit [DBG] from='client.14922 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:36:25.154 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:24 smithi132 bash[18607]: audit 2024-04-25T20:36:23.431047+0000 mgr.x (mgr.14254) 1378 : audit [DBG] from='client.14922 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:36:25.154 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:24 smithi132 bash[18607]: cluster 2024-04-25T20:36:23.547358+0000 mgr.x (mgr.14254) 1379 : cluster [DBG] pgmap v1123: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:25.155 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:24 smithi132 bash[18607]: cluster 2024-04-25T20:36:23.547358+0000 mgr.x (mgr.14254) 1379 : cluster [DBG] pgmap v1123: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:27.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:26 smithi132 bash[18607]: cluster 2024-04-25T20:36:25.548734+0000 mgr.x (mgr.14254) 1380 : cluster [DBG] pgmap v1124: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:27.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:26 smithi132 bash[18607]: cluster 2024-04-25T20:36:25.548734+0000 mgr.x (mgr.14254) 1380 : cluster [DBG] pgmap v1124: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:29.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:28 smithi132 bash[18607]: cluster 2024-04-25T20:36:27.550288+0000 mgr.x (mgr.14254) 1381 : cluster [DBG] pgmap v1125: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:29.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:28 smithi132 bash[18607]: cluster 2024-04-25T20:36:27.550288+0000 mgr.x (mgr.14254) 1381 : cluster [DBG] pgmap v1125: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:29.898 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:36:30.976 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:30 smithi132 bash[18607]: cluster 2024-04-25T20:36:29.552198+0000 mgr.x (mgr.14254) 1382 : cluster [DBG] pgmap v1126: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:30.977 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:30 smithi132 bash[18607]: cluster 2024-04-25T20:36:29.552198+0000 mgr.x (mgr.14254) 1382 : cluster [DBG] pgmap v1126: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:32.662 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:36:32.662 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:36:33.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:32 smithi132 bash[18607]: cluster 2024-04-25T20:36:31.553353+0000 mgr.x (mgr.14254) 1383 : cluster [DBG] pgmap v1127: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:33.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:32 smithi132 bash[18607]: cluster 2024-04-25T20:36:31.553353+0000 mgr.x (mgr.14254) 1383 : cluster [DBG] pgmap v1127: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:33.345 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:36:34.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:33 smithi132 bash[18607]: audit 2024-04-25T20:36:32.660892+0000 mgr.x (mgr.14254) 1384 : audit [DBG] from='client.14924 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:36:34.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:33 smithi132 bash[18607]: audit 2024-04-25T20:36:32.660892+0000 mgr.x (mgr.14254) 1384 : audit [DBG] from='client.14924 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:36:34.346 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:36:35.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:34 smithi132 bash[18607]: cluster 2024-04-25T20:36:33.555211+0000 mgr.x (mgr.14254) 1385 : cluster [DBG] pgmap v1128: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:35.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:34 smithi132 bash[18607]: cluster 2024-04-25T20:36:33.555211+0000 mgr.x (mgr.14254) 1385 : cluster [DBG] pgmap v1128: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:36.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:35 smithi132 bash[18607]: audit 2024-04-25T20:36:34.834328+0000 mon.a (mon.0) 525 : audit [DBG] from='client.? 172.17.0.2:0/2021462918' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:36:36.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:35 smithi132 bash[18607]: audit 2024-04-25T20:36:34.834328+0000 mon.a (mon.0) 525 : audit [DBG] from='client.? 172.17.0.2:0/2021462918' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:36:37.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:36 smithi132 bash[18607]: cluster 2024-04-25T20:36:35.556424+0000 mgr.x (mgr.14254) 1386 : cluster [DBG] pgmap v1129: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:37.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:36 smithi132 bash[18607]: cluster 2024-04-25T20:36:35.556424+0000 mgr.x (mgr.14254) 1386 : cluster [DBG] pgmap v1129: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:39.108 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:36:39.130 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:38 smithi132 bash[18607]: cluster 2024-04-25T20:36:37.557817+0000 mgr.x (mgr.14254) 1387 : cluster [DBG] pgmap v1130: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:39.131 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:38 smithi132 bash[18607]: cluster 2024-04-25T20:36:37.557817+0000 mgr.x (mgr.14254) 1387 : cluster [DBG] pgmap v1130: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:41.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:40 smithi132 bash[18607]: cluster 2024-04-25T20:36:39.559722+0000 mgr.x (mgr.14254) 1388 : cluster [DBG] pgmap v1131: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:41.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:40 smithi132 bash[18607]: cluster 2024-04-25T20:36:39.559722+0000 mgr.x (mgr.14254) 1388 : cluster [DBG] pgmap v1131: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:41.885 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:36:41.885 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:36:42.605 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:36:43.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:42 smithi132 bash[18607]: cluster 2024-04-25T20:36:41.560417+0000 mgr.x (mgr.14254) 1389 : cluster [DBG] pgmap v1132: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:43.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:42 smithi132 bash[18607]: cluster 2024-04-25T20:36:41.560417+0000 mgr.x (mgr.14254) 1389 : cluster [DBG] pgmap v1132: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:43.606 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:36:44.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:43 smithi132 bash[18607]: audit 2024-04-25T20:36:41.885850+0000 mgr.x (mgr.14254) 1390 : audit [DBG] from='client.14928 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:36:44.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:43 smithi132 bash[18607]: audit 2024-04-25T20:36:41.885850+0000 mgr.x (mgr.14254) 1390 : audit [DBG] from='client.14928 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:36:45.141 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:44 smithi132 bash[18607]: cluster 2024-04-25T20:36:43.562210+0000 mgr.x (mgr.14254) 1391 : cluster [DBG] pgmap v1133: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:45.141 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:44 smithi132 bash[18607]: cluster 2024-04-25T20:36:43.562210+0000 mgr.x (mgr.14254) 1391 : cluster [DBG] pgmap v1133: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:47.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:46 smithi132 bash[18607]: cluster 2024-04-25T20:36:45.563617+0000 mgr.x (mgr.14254) 1392 : cluster [DBG] pgmap v1134: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:47.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:46 smithi132 bash[18607]: cluster 2024-04-25T20:36:45.563617+0000 mgr.x (mgr.14254) 1392 : cluster [DBG] pgmap v1134: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:48.365 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:36:49.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:48 smithi132 bash[18607]: cluster 2024-04-25T20:36:47.565082+0000 mgr.x (mgr.14254) 1393 : cluster [DBG] pgmap v1135: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:49.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:48 smithi132 bash[18607]: cluster 2024-04-25T20:36:47.565082+0000 mgr.x (mgr.14254) 1393 : cluster [DBG] pgmap v1135: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:51.196 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:36:51.196 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:36:51.209 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:50 smithi132 bash[18607]: cluster 2024-04-25T20:36:49.566945+0000 mgr.x (mgr.14254) 1394 : cluster [DBG] pgmap v1136: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:51.209 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:50 smithi132 bash[18607]: cluster 2024-04-25T20:36:49.566945+0000 mgr.x (mgr.14254) 1394 : cluster [DBG] pgmap v1136: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:51.960 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:36:52.961 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:36:53.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:52 smithi132 bash[18607]: audit 2024-04-25T20:36:51.197019+0000 mgr.x (mgr.14254) 1395 : audit [DBG] from='client.14930 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:36:53.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:52 smithi132 bash[18607]: audit 2024-04-25T20:36:51.197019+0000 mgr.x (mgr.14254) 1395 : audit [DBG] from='client.14930 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:36:53.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:52 smithi132 bash[18607]: cluster 2024-04-25T20:36:51.568379+0000 mgr.x (mgr.14254) 1396 : cluster [DBG] pgmap v1137: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:53.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:52 smithi132 bash[18607]: cluster 2024-04-25T20:36:51.568379+0000 mgr.x (mgr.14254) 1396 : cluster [DBG] pgmap v1137: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:55.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:54 smithi132 bash[18607]: cluster 2024-04-25T20:36:53.570283+0000 mgr.x (mgr.14254) 1397 : cluster [DBG] pgmap v1138: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:55.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:54 smithi132 bash[18607]: cluster 2024-04-25T20:36:53.570283+0000 mgr.x (mgr.14254) 1397 : cluster [DBG] pgmap v1138: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:57.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:56 smithi132 bash[18607]: cluster 2024-04-25T20:36:55.571709+0000 mgr.x (mgr.14254) 1398 : cluster [DBG] pgmap v1139: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:57.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:56 smithi132 bash[18607]: cluster 2024-04-25T20:36:55.571709+0000 mgr.x (mgr.14254) 1398 : cluster [DBG] pgmap v1139: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:57.724 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:36:59.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:58 smithi132 bash[18607]: cluster 2024-04-25T20:36:57.573119+0000 mgr.x (mgr.14254) 1399 : cluster [DBG] pgmap v1140: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:36:59.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:36:58 smithi132 bash[18607]: cluster 2024-04-25T20:36:57.573119+0000 mgr.x (mgr.14254) 1399 : cluster [DBG] pgmap v1140: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:00.391 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:37:00.391 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:37:01.063 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:37:01.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:00 smithi132 bash[18607]: cluster 2024-04-25T20:36:59.574983+0000 mgr.x (mgr.14254) 1400 : cluster [DBG] pgmap v1141: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:01.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:00 smithi132 bash[18607]: cluster 2024-04-25T20:36:59.574983+0000 mgr.x (mgr.14254) 1400 : cluster [DBG] pgmap v1141: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:02.065 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:37:02.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:01 smithi132 bash[18607]: audit 2024-04-25T20:37:00.392879+0000 mgr.x (mgr.14254) 1401 : audit [DBG] from='client.14932 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:37:02.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:01 smithi132 bash[18607]: audit 2024-04-25T20:37:00.392879+0000 mgr.x (mgr.14254) 1401 : audit [DBG] from='client.14932 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:37:03.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:02 smithi132 bash[18607]: cluster 2024-04-25T20:37:01.576385+0000 mgr.x (mgr.14254) 1402 : cluster [DBG] pgmap v1142: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:03.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:02 smithi132 bash[18607]: cluster 2024-04-25T20:37:01.576385+0000 mgr.x (mgr.14254) 1402 : cluster [DBG] pgmap v1142: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:05.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:04 smithi132 bash[18607]: cluster 2024-04-25T20:37:03.578236+0000 mgr.x (mgr.14254) 1403 : cluster [DBG] pgmap v1143: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:05.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:04 smithi132 bash[18607]: cluster 2024-04-25T20:37:03.578236+0000 mgr.x (mgr.14254) 1403 : cluster [DBG] pgmap v1143: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:06.820 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:37:07.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:06 smithi132 bash[18607]: cluster 2024-04-25T20:37:05.579702+0000 mgr.x (mgr.14254) 1404 : cluster [DBG] pgmap v1144: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:07.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:06 smithi132 bash[18607]: cluster 2024-04-25T20:37:05.579702+0000 mgr.x (mgr.14254) 1404 : cluster [DBG] pgmap v1144: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:09.221 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:08 smithi132 bash[18607]: cluster 2024-04-25T20:37:07.581157+0000 mgr.x (mgr.14254) 1405 : cluster [DBG] pgmap v1145: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:09.221 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:08 smithi132 bash[18607]: cluster 2024-04-25T20:37:07.581157+0000 mgr.x (mgr.14254) 1405 : cluster [DBG] pgmap v1145: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:09.622 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:37:09.622 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:37:10.373 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:37:11.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:10 smithi132 bash[18607]: cluster 2024-04-25T20:37:09.583095+0000 mgr.x (mgr.14254) 1406 : cluster [DBG] pgmap v1146: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:11.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:10 smithi132 bash[18607]: cluster 2024-04-25T20:37:09.583095+0000 mgr.x (mgr.14254) 1406 : cluster [DBG] pgmap v1146: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:11.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:10 smithi132 bash[18607]: audit 2024-04-25T20:37:09.620888+0000 mgr.x (mgr.14254) 1407 : audit [DBG] from='client.14934 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:37:11.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:10 smithi132 bash[18607]: audit 2024-04-25T20:37:09.620888+0000 mgr.x (mgr.14254) 1407 : audit [DBG] from='client.14934 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:37:11.374 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:37:13.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:12 smithi132 bash[18607]: cluster 2024-04-25T20:37:11.583921+0000 mgr.x (mgr.14254) 1408 : cluster [DBG] pgmap v1147: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:13.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:12 smithi132 bash[18607]: cluster 2024-04-25T20:37:11.583921+0000 mgr.x (mgr.14254) 1408 : cluster [DBG] pgmap v1147: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:15.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:14 smithi132 bash[18607]: cluster 2024-04-25T20:37:13.585853+0000 mgr.x (mgr.14254) 1409 : cluster [DBG] pgmap v1148: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:15.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:14 smithi132 bash[18607]: cluster 2024-04-25T20:37:13.585853+0000 mgr.x (mgr.14254) 1409 : cluster [DBG] pgmap v1148: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:16.131 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:37:17.101 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:16 smithi132 bash[18607]: cluster 2024-04-25T20:37:15.587279+0000 mgr.x (mgr.14254) 1410 : cluster [DBG] pgmap v1149: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:17.101 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:16 smithi132 bash[18607]: cluster 2024-04-25T20:37:15.587279+0000 mgr.x (mgr.14254) 1410 : cluster [DBG] pgmap v1149: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:18.982 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:37:18.983 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:37:19.161 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:18 smithi132 bash[18607]: cluster 2024-04-25T20:37:17.588700+0000 mgr.x (mgr.14254) 1411 : cluster [DBG] pgmap v1150: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:19.161 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:18 smithi132 bash[18607]: cluster 2024-04-25T20:37:17.588700+0000 mgr.x (mgr.14254) 1411 : cluster [DBG] pgmap v1150: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:19.711 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:37:20.711 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:37:21.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:20 smithi132 bash[18607]: audit 2024-04-25T20:37:18.980916+0000 mgr.x (mgr.14254) 1412 : audit [DBG] from='client.14936 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:37:21.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:20 smithi132 bash[18607]: audit 2024-04-25T20:37:18.980916+0000 mgr.x (mgr.14254) 1412 : audit [DBG] from='client.14936 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:37:21.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:20 smithi132 bash[18607]: cluster 2024-04-25T20:37:19.590534+0000 mgr.x (mgr.14254) 1413 : cluster [DBG] pgmap v1151: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:21.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:20 smithi132 bash[18607]: cluster 2024-04-25T20:37:19.590534+0000 mgr.x (mgr.14254) 1413 : cluster [DBG] pgmap v1151: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:22.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:21 smithi132 bash[18607]: cluster 2024-04-25T20:37:21.591919+0000 mgr.x (mgr.14254) 1414 : cluster [DBG] pgmap v1152: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:22.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:21 smithi132 bash[18607]: cluster 2024-04-25T20:37:21.591919+0000 mgr.x (mgr.14254) 1414 : cluster [DBG] pgmap v1152: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:25.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:24 smithi132 bash[18607]: cluster 2024-04-25T20:37:23.593801+0000 mgr.x (mgr.14254) 1415 : cluster [DBG] pgmap v1153: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:25.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:24 smithi132 bash[18607]: cluster 2024-04-25T20:37:23.593801+0000 mgr.x (mgr.14254) 1415 : cluster [DBG] pgmap v1153: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:25.470 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:37:26.701 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:26 smithi132 bash[18607]: cluster 2024-04-25T20:37:25.595247+0000 mgr.x (mgr.14254) 1416 : cluster [DBG] pgmap v1154: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:26.701 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:26 smithi132 bash[18607]: cluster 2024-04-25T20:37:25.595247+0000 mgr.x (mgr.14254) 1416 : cluster [DBG] pgmap v1154: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:28.383 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:37:28.383 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:37:29.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:28 smithi132 bash[18607]: cluster 2024-04-25T20:37:27.596704+0000 mgr.x (mgr.14254) 1417 : cluster [DBG] pgmap v1155: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:29.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:28 smithi132 bash[18607]: cluster 2024-04-25T20:37:27.596704+0000 mgr.x (mgr.14254) 1417 : cluster [DBG] pgmap v1155: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:29.108 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:37:30.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:29 smithi132 bash[18607]: audit 2024-04-25T20:37:28.381606+0000 mgr.x (mgr.14254) 1418 : audit [DBG] from='client.14938 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:37:30.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:29 smithi132 bash[18607]: audit 2024-04-25T20:37:28.381606+0000 mgr.x (mgr.14254) 1418 : audit [DBG] from='client.14938 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:37:30.108 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:37:31.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:30 smithi132 bash[18607]: cluster 2024-04-25T20:37:29.598629+0000 mgr.x (mgr.14254) 1419 : cluster [DBG] pgmap v1156: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:31.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:30 smithi132 bash[18607]: cluster 2024-04-25T20:37:29.598629+0000 mgr.x (mgr.14254) 1419 : cluster [DBG] pgmap v1156: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:33.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:32 smithi132 bash[18607]: cluster 2024-04-25T20:37:31.600004+0000 mgr.x (mgr.14254) 1420 : cluster [DBG] pgmap v1157: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:33.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:32 smithi132 bash[18607]: cluster 2024-04-25T20:37:31.600004+0000 mgr.x (mgr.14254) 1420 : cluster [DBG] pgmap v1157: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:34.865 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:37:35.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:34 smithi132 bash[18607]: cluster 2024-04-25T20:37:33.601911+0000 mgr.x (mgr.14254) 1421 : cluster [DBG] pgmap v1158: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:35.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:34 smithi132 bash[18607]: cluster 2024-04-25T20:37:33.601911+0000 mgr.x (mgr.14254) 1421 : cluster [DBG] pgmap v1158: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:37.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:36 smithi132 bash[18607]: cluster 2024-04-25T20:37:35.603331+0000 mgr.x (mgr.14254) 1422 : cluster [DBG] pgmap v1159: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:37.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:36 smithi132 bash[18607]: cluster 2024-04-25T20:37:35.603331+0000 mgr.x (mgr.14254) 1422 : cluster [DBG] pgmap v1159: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:37.611 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:37:37.611 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:37:38.325 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:37:39.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:38 smithi132 bash[18607]: cluster 2024-04-25T20:37:37.604164+0000 mgr.x (mgr.14254) 1423 : cluster [DBG] pgmap v1160: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:39.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:38 smithi132 bash[18607]: cluster 2024-04-25T20:37:37.604164+0000 mgr.x (mgr.14254) 1423 : cluster [DBG] pgmap v1160: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:39.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:38 smithi132 bash[18607]: audit 2024-04-25T20:37:37.609835+0000 mgr.x (mgr.14254) 1424 : audit [DBG] from='client.14940 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:37:39.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:38 smithi132 bash[18607]: audit 2024-04-25T20:37:37.609835+0000 mgr.x (mgr.14254) 1424 : audit [DBG] from='client.14940 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:37:39.326 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:37:41.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:40 smithi132 bash[18607]: cluster 2024-04-25T20:37:39.605993+0000 mgr.x (mgr.14254) 1425 : cluster [DBG] pgmap v1161: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:41.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:40 smithi132 bash[18607]: cluster 2024-04-25T20:37:39.605993+0000 mgr.x (mgr.14254) 1425 : cluster [DBG] pgmap v1161: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:43.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:42 smithi132 bash[18607]: cluster 2024-04-25T20:37:41.607330+0000 mgr.x (mgr.14254) 1426 : cluster [DBG] pgmap v1162: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:43.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:42 smithi132 bash[18607]: cluster 2024-04-25T20:37:41.607330+0000 mgr.x (mgr.14254) 1426 : cluster [DBG] pgmap v1162: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:44.079 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:37:44.947 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:44 smithi132 bash[18607]: cluster 2024-04-25T20:37:43.608448+0000 mgr.x (mgr.14254) 1427 : cluster [DBG] pgmap v1163: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:44.947 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:44 smithi132 bash[18607]: cluster 2024-04-25T20:37:43.608448+0000 mgr.x (mgr.14254) 1427 : cluster [DBG] pgmap v1163: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:46.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:45 smithi132 bash[18607]: audit 2024-04-25T20:37:45.216282+0000 mon.a (mon.0) 526 : audit [DBG] from='client.? 172.17.0.2:0/1106999485' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:37:46.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:45 smithi132 bash[18607]: audit 2024-04-25T20:37:45.216282+0000 mon.a (mon.0) 526 : audit [DBG] from='client.? 172.17.0.2:0/1106999485' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:37:47.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:46 smithi132 bash[18607]: cluster 2024-04-25T20:37:45.609866+0000 mgr.x (mgr.14254) 1428 : cluster [DBG] pgmap v1164: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:47.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:46 smithi132 bash[18607]: cluster 2024-04-25T20:37:45.609866+0000 mgr.x (mgr.14254) 1428 : cluster [DBG] pgmap v1164: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:47.108 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:37:47.108 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:37:47.771 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:37:48.772 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:37:49.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:48 smithi132 bash[18607]: audit 2024-04-25T20:37:47.106350+0000 mgr.x (mgr.14254) 1429 : audit [DBG] from='client.14944 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:37:49.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:48 smithi132 bash[18607]: audit 2024-04-25T20:37:47.106350+0000 mgr.x (mgr.14254) 1429 : audit [DBG] from='client.14944 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:37:49.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:48 smithi132 bash[18607]: cluster 2024-04-25T20:37:47.611310+0000 mgr.x (mgr.14254) 1430 : cluster [DBG] pgmap v1165: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:49.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:48 smithi132 bash[18607]: cluster 2024-04-25T20:37:47.611310+0000 mgr.x (mgr.14254) 1430 : cluster [DBG] pgmap v1165: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:51.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:50 smithi132 bash[18607]: cluster 2024-04-25T20:37:49.613190+0000 mgr.x (mgr.14254) 1431 : cluster [DBG] pgmap v1166: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:51.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:50 smithi132 bash[18607]: cluster 2024-04-25T20:37:49.613190+0000 mgr.x (mgr.14254) 1431 : cluster [DBG] pgmap v1166: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:53.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:52 smithi132 bash[18607]: cluster 2024-04-25T20:37:51.614576+0000 mgr.x (mgr.14254) 1432 : cluster [DBG] pgmap v1167: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:53.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:52 smithi132 bash[18607]: cluster 2024-04-25T20:37:51.614576+0000 mgr.x (mgr.14254) 1432 : cluster [DBG] pgmap v1167: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:53.529 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:37:55.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:54 smithi132 bash[18607]: cluster 2024-04-25T20:37:53.615611+0000 mgr.x (mgr.14254) 1433 : cluster [DBG] pgmap v1168: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:55.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:54 smithi132 bash[18607]: cluster 2024-04-25T20:37:53.615611+0000 mgr.x (mgr.14254) 1433 : cluster [DBG] pgmap v1168: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:56.405 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:37:56.405 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:37:57.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:56 smithi132 bash[18607]: cluster 2024-04-25T20:37:55.616952+0000 mgr.x (mgr.14254) 1434 : cluster [DBG] pgmap v1169: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:57.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:56 smithi132 bash[18607]: cluster 2024-04-25T20:37:55.616952+0000 mgr.x (mgr.14254) 1434 : cluster [DBG] pgmap v1169: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:57.113 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:37:58.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:57 smithi132 bash[18607]: audit 2024-04-25T20:37:56.403853+0000 mgr.x (mgr.14254) 1435 : audit [DBG] from='client.14946 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:37:58.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:57 smithi132 bash[18607]: audit 2024-04-25T20:37:56.403853+0000 mgr.x (mgr.14254) 1435 : audit [DBG] from='client.14946 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:37:58.114 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:37:59.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:58 smithi132 bash[18607]: cluster 2024-04-25T20:37:57.618472+0000 mgr.x (mgr.14254) 1436 : cluster [DBG] pgmap v1170: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:37:59.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:37:58 smithi132 bash[18607]: cluster 2024-04-25T20:37:57.618472+0000 mgr.x (mgr.14254) 1436 : cluster [DBG] pgmap v1170: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:01.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:00 smithi132 bash[18607]: cluster 2024-04-25T20:37:59.620344+0000 mgr.x (mgr.14254) 1437 : cluster [DBG] pgmap v1171: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:01.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:00 smithi132 bash[18607]: cluster 2024-04-25T20:37:59.620344+0000 mgr.x (mgr.14254) 1437 : cluster [DBG] pgmap v1171: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:02.871 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:38:03.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:02 smithi132 bash[18607]: cluster 2024-04-25T20:38:01.621779+0000 mgr.x (mgr.14254) 1438 : cluster [DBG] pgmap v1172: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:03.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:02 smithi132 bash[18607]: cluster 2024-04-25T20:38:01.621779+0000 mgr.x (mgr.14254) 1438 : cluster [DBG] pgmap v1172: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:05.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:04 smithi132 bash[18607]: cluster 2024-04-25T20:38:03.623623+0000 mgr.x (mgr.14254) 1439 : cluster [DBG] pgmap v1173: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:05.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:04 smithi132 bash[18607]: cluster 2024-04-25T20:38:03.623623+0000 mgr.x (mgr.14254) 1439 : cluster [DBG] pgmap v1173: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:05.679 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:38:05.679 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:38:06.368 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:38:07.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:06 smithi132 bash[18607]: cluster 2024-04-25T20:38:05.625047+0000 mgr.x (mgr.14254) 1440 : cluster [DBG] pgmap v1174: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:07.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:06 smithi132 bash[18607]: cluster 2024-04-25T20:38:05.625047+0000 mgr.x (mgr.14254) 1440 : cluster [DBG] pgmap v1174: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:07.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:06 smithi132 bash[18607]: audit 2024-04-25T20:38:05.677994+0000 mgr.x (mgr.14254) 1441 : audit [DBG] from='client.14948 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:38:07.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:06 smithi132 bash[18607]: audit 2024-04-25T20:38:05.677994+0000 mgr.x (mgr.14254) 1441 : audit [DBG] from='client.14948 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:38:07.369 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:38:09.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:08 smithi132 bash[18607]: cluster 2024-04-25T20:38:07.626188+0000 mgr.x (mgr.14254) 1442 : cluster [DBG] pgmap v1175: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:09.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:08 smithi132 bash[18607]: cluster 2024-04-25T20:38:07.626188+0000 mgr.x (mgr.14254) 1442 : cluster [DBG] pgmap v1175: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:11.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:10 smithi132 bash[18607]: cluster 2024-04-25T20:38:09.628202+0000 mgr.x (mgr.14254) 1443 : cluster [DBG] pgmap v1176: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:11.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:10 smithi132 bash[18607]: cluster 2024-04-25T20:38:09.628202+0000 mgr.x (mgr.14254) 1443 : cluster [DBG] pgmap v1176: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:12.129 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:38:12.961 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:12 smithi132 bash[18607]: cluster 2024-04-25T20:38:11.629473+0000 mgr.x (mgr.14254) 1444 : cluster [DBG] pgmap v1177: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:12.961 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:12 smithi132 bash[18607]: cluster 2024-04-25T20:38:11.629473+0000 mgr.x (mgr.14254) 1444 : cluster [DBG] pgmap v1177: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:14.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:14 smithi132 bash[18607]: cluster 2024-04-25T20:38:13.631392+0000 mgr.x (mgr.14254) 1445 : cluster [DBG] pgmap v1178: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:14.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:14 smithi132 bash[18607]: cluster 2024-04-25T20:38:13.631392+0000 mgr.x (mgr.14254) 1445 : cluster [DBG] pgmap v1178: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:14.887 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:38:14.887 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:38:15.629 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:38:16.630 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:38:17.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:16 smithi132 bash[18607]: audit 2024-04-25T20:38:14.888622+0000 mgr.x (mgr.14254) 1446 : audit [DBG] from='client.14950 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:38:17.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:16 smithi132 bash[18607]: audit 2024-04-25T20:38:14.888622+0000 mgr.x (mgr.14254) 1446 : audit [DBG] from='client.14950 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:38:17.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:16 smithi132 bash[18607]: cluster 2024-04-25T20:38:15.633067+0000 mgr.x (mgr.14254) 1447 : cluster [DBG] pgmap v1179: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:17.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:16 smithi132 bash[18607]: cluster 2024-04-25T20:38:15.633067+0000 mgr.x (mgr.14254) 1447 : cluster [DBG] pgmap v1179: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:19.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:18 smithi132 bash[18607]: cluster 2024-04-25T20:38:17.634465+0000 mgr.x (mgr.14254) 1448 : cluster [DBG] pgmap v1180: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:19.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:18 smithi132 bash[18607]: cluster 2024-04-25T20:38:17.634465+0000 mgr.x (mgr.14254) 1448 : cluster [DBG] pgmap v1180: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:21.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:20 smithi132 bash[18607]: cluster 2024-04-25T20:38:19.636373+0000 mgr.x (mgr.14254) 1449 : cluster [DBG] pgmap v1181: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:21.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:20 smithi132 bash[18607]: cluster 2024-04-25T20:38:19.636373+0000 mgr.x (mgr.14254) 1449 : cluster [DBG] pgmap v1181: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:21.390 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:38:22.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:22 smithi132 bash[18607]: cluster 2024-04-25T20:38:21.637814+0000 mgr.x (mgr.14254) 1450 : cluster [DBG] pgmap v1182: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:22.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:22 smithi132 bash[18607]: cluster 2024-04-25T20:38:21.637814+0000 mgr.x (mgr.14254) 1450 : cluster [DBG] pgmap v1182: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:24.330 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:38:24.330 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:38:25.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:24 smithi132 bash[18607]: cluster 2024-04-25T20:38:23.639647+0000 mgr.x (mgr.14254) 1451 : cluster [DBG] pgmap v1183: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:25.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:24 smithi132 bash[18607]: cluster 2024-04-25T20:38:23.639647+0000 mgr.x (mgr.14254) 1451 : cluster [DBG] pgmap v1183: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:25.032 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:38:26.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:25 smithi132 bash[18607]: audit 2024-04-25T20:38:24.331263+0000 mgr.x (mgr.14254) 1452 : audit [DBG] from='client.14952 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:38:26.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:25 smithi132 bash[18607]: audit 2024-04-25T20:38:24.331263+0000 mgr.x (mgr.14254) 1452 : audit [DBG] from='client.14952 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:38:26.033 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:38:27.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:26 smithi132 bash[18607]: cluster 2024-04-25T20:38:25.641046+0000 mgr.x (mgr.14254) 1453 : cluster [DBG] pgmap v1184: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:27.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:26 smithi132 bash[18607]: cluster 2024-04-25T20:38:25.641046+0000 mgr.x (mgr.14254) 1453 : cluster [DBG] pgmap v1184: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:29.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:28 smithi132 bash[18607]: cluster 2024-04-25T20:38:27.642422+0000 mgr.x (mgr.14254) 1454 : cluster [DBG] pgmap v1185: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:29.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:28 smithi132 bash[18607]: cluster 2024-04-25T20:38:27.642422+0000 mgr.x (mgr.14254) 1454 : cluster [DBG] pgmap v1185: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:30.795 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:38:31.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:30 smithi132 bash[18607]: cluster 2024-04-25T20:38:29.644318+0000 mgr.x (mgr.14254) 1455 : cluster [DBG] pgmap v1186: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:31.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:30 smithi132 bash[18607]: cluster 2024-04-25T20:38:29.644318+0000 mgr.x (mgr.14254) 1455 : cluster [DBG] pgmap v1186: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:33.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:32 smithi132 bash[18607]: cluster 2024-04-25T20:38:31.645091+0000 mgr.x (mgr.14254) 1456 : cluster [DBG] pgmap v1187: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:33.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:32 smithi132 bash[18607]: cluster 2024-04-25T20:38:31.645091+0000 mgr.x (mgr.14254) 1456 : cluster [DBG] pgmap v1187: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:33.490 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:38:33.490 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:38:34.248 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:38:35.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:34 smithi132 bash[18607]: audit 2024-04-25T20:38:33.491751+0000 mgr.x (mgr.14254) 1457 : audit [DBG] from='client.14954 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:38:35.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:34 smithi132 bash[18607]: audit 2024-04-25T20:38:33.491751+0000 mgr.x (mgr.14254) 1457 : audit [DBG] from='client.14954 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:38:35.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:34 smithi132 bash[18607]: cluster 2024-04-25T20:38:33.646792+0000 mgr.x (mgr.14254) 1458 : cluster [DBG] pgmap v1188: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:35.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:34 smithi132 bash[18607]: cluster 2024-04-25T20:38:33.646792+0000 mgr.x (mgr.14254) 1458 : cluster [DBG] pgmap v1188: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:35.249 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:38:37.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:36 smithi132 bash[18607]: cluster 2024-04-25T20:38:35.648354+0000 mgr.x (mgr.14254) 1459 : cluster [DBG] pgmap v1189: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:37.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:36 smithi132 bash[18607]: cluster 2024-04-25T20:38:35.648354+0000 mgr.x (mgr.14254) 1459 : cluster [DBG] pgmap v1189: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:39.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:38 smithi132 bash[18607]: cluster 2024-04-25T20:38:37.649965+0000 mgr.x (mgr.14254) 1460 : cluster [DBG] pgmap v1190: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:39.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:38 smithi132 bash[18607]: cluster 2024-04-25T20:38:37.649965+0000 mgr.x (mgr.14254) 1460 : cluster [DBG] pgmap v1190: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:40.010 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:38:41.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:40 smithi132 bash[18607]: cluster 2024-04-25T20:38:39.651901+0000 mgr.x (mgr.14254) 1461 : cluster [DBG] pgmap v1191: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:41.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:40 smithi132 bash[18607]: cluster 2024-04-25T20:38:39.651901+0000 mgr.x (mgr.14254) 1461 : cluster [DBG] pgmap v1191: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:42.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:42 smithi132 bash[18607]: cluster 2024-04-25T20:38:41.653366+0000 mgr.x (mgr.14254) 1462 : cluster [DBG] pgmap v1192: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:42.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:42 smithi132 bash[18607]: cluster 2024-04-25T20:38:41.653366+0000 mgr.x (mgr.14254) 1462 : cluster [DBG] pgmap v1192: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:42.935 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:38:42.935 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:38:43.635 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:38:44.636 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:38:45.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:44 smithi132 bash[18607]: audit 2024-04-25T20:38:42.934641+0000 mgr.x (mgr.14254) 1463 : audit [DBG] from='client.14956 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:38:45.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:44 smithi132 bash[18607]: audit 2024-04-25T20:38:42.934641+0000 mgr.x (mgr.14254) 1463 : audit [DBG] from='client.14956 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:38:45.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:44 smithi132 bash[18607]: cluster 2024-04-25T20:38:43.655252+0000 mgr.x (mgr.14254) 1464 : cluster [DBG] pgmap v1193: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:45.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:44 smithi132 bash[18607]: cluster 2024-04-25T20:38:43.655252+0000 mgr.x (mgr.14254) 1464 : cluster [DBG] pgmap v1193: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:47.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:46 smithi132 bash[18607]: cluster 2024-04-25T20:38:45.656795+0000 mgr.x (mgr.14254) 1465 : cluster [DBG] pgmap v1194: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:47.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:46 smithi132 bash[18607]: cluster 2024-04-25T20:38:45.656795+0000 mgr.x (mgr.14254) 1465 : cluster [DBG] pgmap v1194: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:49.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:48 smithi132 bash[18607]: cluster 2024-04-25T20:38:47.658224+0000 mgr.x (mgr.14254) 1466 : cluster [DBG] pgmap v1195: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:49.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:48 smithi132 bash[18607]: cluster 2024-04-25T20:38:47.658224+0000 mgr.x (mgr.14254) 1466 : cluster [DBG] pgmap v1195: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:49.397 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:38:50.770 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:50 smithi132 bash[18607]: cluster 2024-04-25T20:38:49.660067+0000 mgr.x (mgr.14254) 1467 : cluster [DBG] pgmap v1196: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:50.771 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:50 smithi132 bash[18607]: cluster 2024-04-25T20:38:49.660067+0000 mgr.x (mgr.14254) 1467 : cluster [DBG] pgmap v1196: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:52.106 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:38:52.106 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:38:52.878 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:38:53.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:52 smithi132 bash[18607]: cluster 2024-04-25T20:38:51.661504+0000 mgr.x (mgr.14254) 1468 : cluster [DBG] pgmap v1197: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:53.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:52 smithi132 bash[18607]: cluster 2024-04-25T20:38:51.661504+0000 mgr.x (mgr.14254) 1468 : cluster [DBG] pgmap v1197: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:53.880 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:38:54.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:53 smithi132 bash[18607]: audit 2024-04-25T20:38:52.105152+0000 mgr.x (mgr.14254) 1469 : audit [DBG] from='client.14958 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:38:54.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:53 smithi132 bash[18607]: audit 2024-04-25T20:38:52.105152+0000 mgr.x (mgr.14254) 1469 : audit [DBG] from='client.14958 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:38:55.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:54 smithi132 bash[18607]: cluster 2024-04-25T20:38:53.663312+0000 mgr.x (mgr.14254) 1470 : cluster [DBG] pgmap v1198: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:55.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:54 smithi132 bash[18607]: cluster 2024-04-25T20:38:53.663312+0000 mgr.x (mgr.14254) 1470 : cluster [DBG] pgmap v1198: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:56.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:55 smithi132 bash[18607]: audit 2024-04-25T20:38:55.606588+0000 mon.a (mon.0) 527 : audit [DBG] from='client.? 172.17.0.2:0/1899903676' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:38:56.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:55 smithi132 bash[18607]: audit 2024-04-25T20:38:55.606588+0000 mon.a (mon.0) 527 : audit [DBG] from='client.? 172.17.0.2:0/1899903676' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:38:57.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:56 smithi132 bash[18607]: cluster 2024-04-25T20:38:55.664192+0000 mgr.x (mgr.14254) 1471 : cluster [DBG] pgmap v1199: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:57.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:56 smithi132 bash[18607]: cluster 2024-04-25T20:38:55.664192+0000 mgr.x (mgr.14254) 1471 : cluster [DBG] pgmap v1199: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:58.648 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:38:59.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:58 smithi132 bash[18607]: cluster 2024-04-25T20:38:57.665584+0000 mgr.x (mgr.14254) 1472 : cluster [DBG] pgmap v1200: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:38:59.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:38:58 smithi132 bash[18607]: cluster 2024-04-25T20:38:57.665584+0000 mgr.x (mgr.14254) 1472 : cluster [DBG] pgmap v1200: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:01.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:00 smithi132 bash[18607]: cluster 2024-04-25T20:38:59.666569+0000 mgr.x (mgr.14254) 1473 : cluster [DBG] pgmap v1201: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:01.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:00 smithi132 bash[18607]: cluster 2024-04-25T20:38:59.666569+0000 mgr.x (mgr.14254) 1473 : cluster [DBG] pgmap v1201: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:01.597 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:39:01.597 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:39:02.317 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:39:03.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:02 smithi132 bash[18607]: audit 2024-04-25T20:39:01.596074+0000 mgr.x (mgr.14254) 1474 : audit [DBG] from='client.14962 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:39:03.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:02 smithi132 bash[18607]: audit 2024-04-25T20:39:01.596074+0000 mgr.x (mgr.14254) 1474 : audit [DBG] from='client.14962 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:39:03.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:02 smithi132 bash[18607]: cluster 2024-04-25T20:39:01.667309+0000 mgr.x (mgr.14254) 1475 : cluster [DBG] pgmap v1202: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:03.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:02 smithi132 bash[18607]: cluster 2024-04-25T20:39:01.667309+0000 mgr.x (mgr.14254) 1475 : cluster [DBG] pgmap v1202: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:03.317 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:39:05.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:04 smithi132 bash[18607]: cluster 2024-04-25T20:39:03.669260+0000 mgr.x (mgr.14254) 1476 : cluster [DBG] pgmap v1203: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:05.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:04 smithi132 bash[18607]: cluster 2024-04-25T20:39:03.669260+0000 mgr.x (mgr.14254) 1476 : cluster [DBG] pgmap v1203: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:07.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:06 smithi132 bash[18607]: cluster 2024-04-25T20:39:05.670639+0000 mgr.x (mgr.14254) 1477 : cluster [DBG] pgmap v1204: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:07.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:06 smithi132 bash[18607]: cluster 2024-04-25T20:39:05.670639+0000 mgr.x (mgr.14254) 1477 : cluster [DBG] pgmap v1204: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:08.074 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:39:09.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:08 smithi132 bash[18607]: cluster 2024-04-25T20:39:07.672209+0000 mgr.x (mgr.14254) 1478 : cluster [DBG] pgmap v1205: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:09.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:08 smithi132 bash[18607]: cluster 2024-04-25T20:39:07.672209+0000 mgr.x (mgr.14254) 1478 : cluster [DBG] pgmap v1205: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:11.004 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:39:11.004 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:39:11.017 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:10 smithi132 bash[18607]: cluster 2024-04-25T20:39:09.674183+0000 mgr.x (mgr.14254) 1479 : cluster [DBG] pgmap v1206: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:11.017 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:10 smithi132 bash[18607]: cluster 2024-04-25T20:39:09.674183+0000 mgr.x (mgr.14254) 1479 : cluster [DBG] pgmap v1206: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:11.660 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:39:12.661 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:39:13.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:12 smithi132 bash[18607]: audit 2024-04-25T20:39:11.005765+0000 mgr.x (mgr.14254) 1480 : audit [DBG] from='client.14964 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:39:13.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:12 smithi132 bash[18607]: audit 2024-04-25T20:39:11.005765+0000 mgr.x (mgr.14254) 1480 : audit [DBG] from='client.14964 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:39:13.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:12 smithi132 bash[18607]: cluster 2024-04-25T20:39:11.675356+0000 mgr.x (mgr.14254) 1481 : cluster [DBG] pgmap v1207: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:13.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:12 smithi132 bash[18607]: cluster 2024-04-25T20:39:11.675356+0000 mgr.x (mgr.14254) 1481 : cluster [DBG] pgmap v1207: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:15.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:14 smithi132 bash[18607]: cluster 2024-04-25T20:39:13.677883+0000 mgr.x (mgr.14254) 1482 : cluster [DBG] pgmap v1208: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:15.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:14 smithi132 bash[18607]: cluster 2024-04-25T20:39:13.677883+0000 mgr.x (mgr.14254) 1482 : cluster [DBG] pgmap v1208: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:17.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:16 smithi132 bash[18607]: cluster 2024-04-25T20:39:15.679329+0000 mgr.x (mgr.14254) 1483 : cluster [DBG] pgmap v1209: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:17.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:16 smithi132 bash[18607]: cluster 2024-04-25T20:39:15.679329+0000 mgr.x (mgr.14254) 1483 : cluster [DBG] pgmap v1209: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:17.415 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:39:19.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:18 smithi132 bash[18607]: cluster 2024-04-25T20:39:17.680793+0000 mgr.x (mgr.14254) 1484 : cluster [DBG] pgmap v1210: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:19.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:18 smithi132 bash[18607]: cluster 2024-04-25T20:39:17.680793+0000 mgr.x (mgr.14254) 1484 : cluster [DBG] pgmap v1210: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:20.193 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:39:20.193 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:39:20.928 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:39:21.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:20 smithi132 bash[18607]: cluster 2024-04-25T20:39:19.682704+0000 mgr.x (mgr.14254) 1485 : cluster [DBG] pgmap v1211: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:21.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:20 smithi132 bash[18607]: cluster 2024-04-25T20:39:19.682704+0000 mgr.x (mgr.14254) 1485 : cluster [DBG] pgmap v1211: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:21.929 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:39:22.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:21 smithi132 bash[18607]: audit 2024-04-25T20:39:20.192381+0000 mgr.x (mgr.14254) 1486 : audit [DBG] from='client.14966 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:39:22.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:21 smithi132 bash[18607]: audit 2024-04-25T20:39:20.192381+0000 mgr.x (mgr.14254) 1486 : audit [DBG] from='client.14966 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:39:23.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:22 smithi132 bash[18607]: cluster 2024-04-25T20:39:21.684083+0000 mgr.x (mgr.14254) 1487 : cluster [DBG] pgmap v1212: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:23.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:22 smithi132 bash[18607]: cluster 2024-04-25T20:39:21.684083+0000 mgr.x (mgr.14254) 1487 : cluster [DBG] pgmap v1212: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:25.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:24 smithi132 bash[18607]: cluster 2024-04-25T20:39:23.685976+0000 mgr.x (mgr.14254) 1488 : cluster [DBG] pgmap v1213: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:25.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:24 smithi132 bash[18607]: cluster 2024-04-25T20:39:23.685976+0000 mgr.x (mgr.14254) 1488 : cluster [DBG] pgmap v1213: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:26.687 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:39:27.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:26 smithi132 bash[18607]: cluster 2024-04-25T20:39:25.687396+0000 mgr.x (mgr.14254) 1489 : cluster [DBG] pgmap v1214: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:27.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:26 smithi132 bash[18607]: cluster 2024-04-25T20:39:25.687396+0000 mgr.x (mgr.14254) 1489 : cluster [DBG] pgmap v1214: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:29.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:28 smithi132 bash[18607]: cluster 2024-04-25T20:39:27.688363+0000 mgr.x (mgr.14254) 1490 : cluster [DBG] pgmap v1215: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:29.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:28 smithi132 bash[18607]: cluster 2024-04-25T20:39:27.688363+0000 mgr.x (mgr.14254) 1490 : cluster [DBG] pgmap v1215: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:29.419 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:39:29.419 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:39:30.115 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:39:31.116 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:39:31.128 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:30 smithi132 bash[18607]: audit 2024-04-25T20:39:29.417384+0000 mgr.x (mgr.14254) 1491 : audit [DBG] from='client.14968 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:39:31.128 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:30 smithi132 bash[18607]: audit 2024-04-25T20:39:29.417384+0000 mgr.x (mgr.14254) 1491 : audit [DBG] from='client.14968 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:39:31.128 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:30 smithi132 bash[18607]: cluster 2024-04-25T20:39:29.690332+0000 mgr.x (mgr.14254) 1492 : cluster [DBG] pgmap v1216: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:31.128 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:30 smithi132 bash[18607]: cluster 2024-04-25T20:39:29.690332+0000 mgr.x (mgr.14254) 1492 : cluster [DBG] pgmap v1216: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:33.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:32 smithi132 bash[18607]: cluster 2024-04-25T20:39:31.691767+0000 mgr.x (mgr.14254) 1493 : cluster [DBG] pgmap v1217: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:33.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:32 smithi132 bash[18607]: cluster 2024-04-25T20:39:31.691767+0000 mgr.x (mgr.14254) 1493 : cluster [DBG] pgmap v1217: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:35.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:34 smithi132 bash[18607]: cluster 2024-04-25T20:39:33.693605+0000 mgr.x (mgr.14254) 1494 : cluster [DBG] pgmap v1218: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:35.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:34 smithi132 bash[18607]: cluster 2024-04-25T20:39:33.693605+0000 mgr.x (mgr.14254) 1494 : cluster [DBG] pgmap v1218: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:35.872 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:39:37.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:36 smithi132 bash[18607]: cluster 2024-04-25T20:39:35.695044+0000 mgr.x (mgr.14254) 1495 : cluster [DBG] pgmap v1219: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:37.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:36 smithi132 bash[18607]: cluster 2024-04-25T20:39:35.695044+0000 mgr.x (mgr.14254) 1495 : cluster [DBG] pgmap v1219: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:38.701 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:39:38.701 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:39:39.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:38 smithi132 bash[18607]: cluster 2024-04-25T20:39:37.696487+0000 mgr.x (mgr.14254) 1496 : cluster [DBG] pgmap v1220: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:39.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:38 smithi132 bash[18607]: cluster 2024-04-25T20:39:37.696487+0000 mgr.x (mgr.14254) 1496 : cluster [DBG] pgmap v1220: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:39.404 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:39:40.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:39 smithi132 bash[18607]: audit 2024-04-25T20:39:38.699499+0000 mgr.x (mgr.14254) 1497 : audit [DBG] from='client.14970 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:39:40.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:39 smithi132 bash[18607]: audit 2024-04-25T20:39:38.699499+0000 mgr.x (mgr.14254) 1497 : audit [DBG] from='client.14970 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:39:40.405 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:39:41.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:40 smithi132 bash[18607]: cluster 2024-04-25T20:39:39.698373+0000 mgr.x (mgr.14254) 1498 : cluster [DBG] pgmap v1221: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:41.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:40 smithi132 bash[18607]: cluster 2024-04-25T20:39:39.698373+0000 mgr.x (mgr.14254) 1498 : cluster [DBG] pgmap v1221: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:43.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:42 smithi132 bash[18607]: cluster 2024-04-25T20:39:41.699801+0000 mgr.x (mgr.14254) 1499 : cluster [DBG] pgmap v1222: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:43.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:42 smithi132 bash[18607]: cluster 2024-04-25T20:39:41.699801+0000 mgr.x (mgr.14254) 1499 : cluster [DBG] pgmap v1222: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:45.167 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:39:45.188 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:44 smithi132 bash[18607]: cluster 2024-04-25T20:39:43.701723+0000 mgr.x (mgr.14254) 1500 : cluster [DBG] pgmap v1223: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:45.188 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:44 smithi132 bash[18607]: cluster 2024-04-25T20:39:43.701723+0000 mgr.x (mgr.14254) 1500 : cluster [DBG] pgmap v1223: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:47.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:46 smithi132 bash[18607]: cluster 2024-04-25T20:39:45.703302+0000 mgr.x (mgr.14254) 1501 : cluster [DBG] pgmap v1224: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:47.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:46 smithi132 bash[18607]: cluster 2024-04-25T20:39:45.703302+0000 mgr.x (mgr.14254) 1501 : cluster [DBG] pgmap v1224: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:47.967 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:39:47.967 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:39:48.652 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:39:49.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:48 smithi132 bash[18607]: cluster 2024-04-25T20:39:47.704055+0000 mgr.x (mgr.14254) 1502 : cluster [DBG] pgmap v1225: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:49.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:48 smithi132 bash[18607]: cluster 2024-04-25T20:39:47.704055+0000 mgr.x (mgr.14254) 1502 : cluster [DBG] pgmap v1225: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:49.653 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:39:50.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:49 smithi132 bash[18607]: audit 2024-04-25T20:39:47.965422+0000 mgr.x (mgr.14254) 1503 : audit [DBG] from='client.14972 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:39:50.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:49 smithi132 bash[18607]: audit 2024-04-25T20:39:47.965422+0000 mgr.x (mgr.14254) 1503 : audit [DBG] from='client.14972 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:39:51.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:50 smithi132 bash[18607]: cluster 2024-04-25T20:39:49.705885+0000 mgr.x (mgr.14254) 1504 : cluster [DBG] pgmap v1226: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:51.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:50 smithi132 bash[18607]: cluster 2024-04-25T20:39:49.705885+0000 mgr.x (mgr.14254) 1504 : cluster [DBG] pgmap v1226: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:53.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:52 smithi132 bash[18607]: cluster 2024-04-25T20:39:51.707286+0000 mgr.x (mgr.14254) 1505 : cluster [DBG] pgmap v1227: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:53.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:52 smithi132 bash[18607]: cluster 2024-04-25T20:39:51.707286+0000 mgr.x (mgr.14254) 1505 : cluster [DBG] pgmap v1227: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:54.412 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:39:55.201 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:54 smithi132 bash[18607]: cluster 2024-04-25T20:39:53.709048+0000 mgr.x (mgr.14254) 1506 : cluster [DBG] pgmap v1228: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:55.201 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:54 smithi132 bash[18607]: cluster 2024-04-25T20:39:53.709048+0000 mgr.x (mgr.14254) 1506 : cluster [DBG] pgmap v1228: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:57.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:56 smithi132 bash[18607]: cluster 2024-04-25T20:39:55.710389+0000 mgr.x (mgr.14254) 1507 : cluster [DBG] pgmap v1229: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:57.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:56 smithi132 bash[18607]: cluster 2024-04-25T20:39:55.710389+0000 mgr.x (mgr.14254) 1507 : cluster [DBG] pgmap v1229: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:57.180 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:39:57.180 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:39:57.867 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:39:58.869 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:39:59.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:58 smithi132 bash[18607]: audit 2024-04-25T20:39:57.180533+0000 mgr.x (mgr.14254) 1508 : audit [DBG] from='client.14974 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:39:59.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:58 smithi132 bash[18607]: audit 2024-04-25T20:39:57.180533+0000 mgr.x (mgr.14254) 1508 : audit [DBG] from='client.14974 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:39:59.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:58 smithi132 bash[18607]: cluster 2024-04-25T20:39:57.711781+0000 mgr.x (mgr.14254) 1509 : cluster [DBG] pgmap v1230: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:39:59.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:39:58 smithi132 bash[18607]: cluster 2024-04-25T20:39:57.711781+0000 mgr.x (mgr.14254) 1509 : cluster [DBG] pgmap v1230: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:01.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:00 smithi132 bash[18607]: cluster 2024-04-25T20:39:59.713677+0000 mgr.x (mgr.14254) 1510 : cluster [DBG] pgmap v1231: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:01.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:00 smithi132 bash[18607]: cluster 2024-04-25T20:39:59.713677+0000 mgr.x (mgr.14254) 1510 : cluster [DBG] pgmap v1231: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:03.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:02 smithi132 bash[18607]: cluster 2024-04-25T20:40:01.715087+0000 mgr.x (mgr.14254) 1511 : cluster [DBG] pgmap v1232: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:03.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:02 smithi132 bash[18607]: cluster 2024-04-25T20:40:01.715087+0000 mgr.x (mgr.14254) 1511 : cluster [DBG] pgmap v1232: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:03.629 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:40:05.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:04 smithi132 bash[18607]: cluster 2024-04-25T20:40:03.716991+0000 mgr.x (mgr.14254) 1512 : cluster [DBG] pgmap v1233: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:05.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:04 smithi132 bash[18607]: cluster 2024-04-25T20:40:03.716991+0000 mgr.x (mgr.14254) 1512 : cluster [DBG] pgmap v1233: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:06.492 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:40:06.492 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:40:07.222 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:06 smithi132 bash[18607]: cluster 2024-04-25T20:40:05.718517+0000 mgr.x (mgr.14254) 1513 : cluster [DBG] pgmap v1234: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:07.222 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:06 smithi132 bash[18607]: cluster 2024-04-25T20:40:05.718517+0000 mgr.x (mgr.14254) 1513 : cluster [DBG] pgmap v1234: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:07.222 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:06 smithi132 bash[18607]: audit 2024-04-25T20:40:05.999839+0000 mon.a (mon.0) 528 : audit [DBG] from='client.? 172.17.0.2:0/734074856' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:40:07.222 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:06 smithi132 bash[18607]: audit 2024-04-25T20:40:05.999839+0000 mon.a (mon.0) 528 : audit [DBG] from='client.? 172.17.0.2:0/734074856' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:40:07.223 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:40:08.225 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:40:08.237 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:07 smithi132 bash[18607]: audit 2024-04-25T20:40:06.493884+0000 mgr.x (mgr.14254) 1514 : audit [DBG] from='client.14978 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:40:08.237 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:07 smithi132 bash[18607]: audit 2024-04-25T20:40:06.493884+0000 mgr.x (mgr.14254) 1514 : audit [DBG] from='client.14978 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:40:09.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:08 smithi132 bash[18607]: cluster 2024-04-25T20:40:07.719964+0000 mgr.x (mgr.14254) 1515 : cluster [DBG] pgmap v1235: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:09.307 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:08 smithi132 bash[18607]: cluster 2024-04-25T20:40:07.719964+0000 mgr.x (mgr.14254) 1515 : cluster [DBG] pgmap v1235: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:11.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:10 smithi132 bash[18607]: cluster 2024-04-25T20:40:09.721829+0000 mgr.x (mgr.14254) 1516 : cluster [DBG] pgmap v1236: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:11.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:10 smithi132 bash[18607]: cluster 2024-04-25T20:40:09.721829+0000 mgr.x (mgr.14254) 1516 : cluster [DBG] pgmap v1236: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:12.988 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:40:13.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:12 smithi132 bash[18607]: cluster 2024-04-25T20:40:11.723224+0000 mgr.x (mgr.14254) 1517 : cluster [DBG] pgmap v1237: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:13.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:12 smithi132 bash[18607]: cluster 2024-04-25T20:40:11.723224+0000 mgr.x (mgr.14254) 1517 : cluster [DBG] pgmap v1237: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:15.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:14 smithi132 bash[18607]: cluster 2024-04-25T20:40:13.725087+0000 mgr.x (mgr.14254) 1518 : cluster [DBG] pgmap v1238: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:15.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:14 smithi132 bash[18607]: cluster 2024-04-25T20:40:13.725087+0000 mgr.x (mgr.14254) 1518 : cluster [DBG] pgmap v1238: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:15.845 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:40:15.845 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:40:16.642 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:40:17.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:16 smithi132 bash[18607]: cluster 2024-04-25T20:40:15.725999+0000 mgr.x (mgr.14254) 1519 : cluster [DBG] pgmap v1239: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:17.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:16 smithi132 bash[18607]: cluster 2024-04-25T20:40:15.725999+0000 mgr.x (mgr.14254) 1519 : cluster [DBG] pgmap v1239: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:17.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:16 smithi132 bash[18607]: audit 2024-04-25T20:40:15.844048+0000 mgr.x (mgr.14254) 1520 : audit [DBG] from='client.14980 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:40:17.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:16 smithi132 bash[18607]: audit 2024-04-25T20:40:15.844048+0000 mgr.x (mgr.14254) 1520 : audit [DBG] from='client.14980 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:40:17.643 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:40:19.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:18 smithi132 bash[18607]: cluster 2024-04-25T20:40:17.727307+0000 mgr.x (mgr.14254) 1521 : cluster [DBG] pgmap v1240: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:19.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:18 smithi132 bash[18607]: cluster 2024-04-25T20:40:17.727307+0000 mgr.x (mgr.14254) 1521 : cluster [DBG] pgmap v1240: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:21.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:20 smithi132 bash[18607]: cluster 2024-04-25T20:40:19.729186+0000 mgr.x (mgr.14254) 1522 : cluster [DBG] pgmap v1241: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:21.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:20 smithi132 bash[18607]: cluster 2024-04-25T20:40:19.729186+0000 mgr.x (mgr.14254) 1522 : cluster [DBG] pgmap v1241: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:22.404 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:40:23.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:22 smithi132 bash[18607]: cluster 2024-04-25T20:40:21.730787+0000 mgr.x (mgr.14254) 1523 : cluster [DBG] pgmap v1242: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:23.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:22 smithi132 bash[18607]: cluster 2024-04-25T20:40:21.730787+0000 mgr.x (mgr.14254) 1523 : cluster [DBG] pgmap v1242: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:25.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:24 smithi132 bash[18607]: cluster 2024-04-25T20:40:23.731873+0000 mgr.x (mgr.14254) 1524 : cluster [DBG] pgmap v1243: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:25.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:24 smithi132 bash[18607]: cluster 2024-04-25T20:40:23.731873+0000 mgr.x (mgr.14254) 1524 : cluster [DBG] pgmap v1243: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:25.462 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:40:25.463 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:40:26.181 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:40:27.182 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:40:27.193 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:26 smithi132 bash[18607]: audit 2024-04-25T20:40:25.464619+0000 mgr.x (mgr.14254) 1525 : audit [DBG] from='client.14982 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:40:27.193 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:26 smithi132 bash[18607]: audit 2024-04-25T20:40:25.464619+0000 mgr.x (mgr.14254) 1525 : audit [DBG] from='client.14982 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:40:27.193 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:26 smithi132 bash[18607]: cluster 2024-04-25T20:40:25.733244+0000 mgr.x (mgr.14254) 1526 : cluster [DBG] pgmap v1244: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:27.193 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:26 smithi132 bash[18607]: cluster 2024-04-25T20:40:25.733244+0000 mgr.x (mgr.14254) 1526 : cluster [DBG] pgmap v1244: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:29.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:28 smithi132 bash[18607]: cluster 2024-04-25T20:40:27.734637+0000 mgr.x (mgr.14254) 1527 : cluster [DBG] pgmap v1245: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:29.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:28 smithi132 bash[18607]: cluster 2024-04-25T20:40:27.734637+0000 mgr.x (mgr.14254) 1527 : cluster [DBG] pgmap v1245: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:31.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:30 smithi132 bash[18607]: cluster 2024-04-25T20:40:29.737179+0000 mgr.x (mgr.14254) 1528 : cluster [DBG] pgmap v1246: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:31.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:30 smithi132 bash[18607]: cluster 2024-04-25T20:40:29.737179+0000 mgr.x (mgr.14254) 1528 : cluster [DBG] pgmap v1246: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:31.942 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:40:33.113 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:32 smithi132 bash[18607]: cluster 2024-04-25T20:40:31.738623+0000 mgr.x (mgr.14254) 1529 : cluster [DBG] pgmap v1247: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:33.114 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:32 smithi132 bash[18607]: cluster 2024-04-25T20:40:31.738623+0000 mgr.x (mgr.14254) 1529 : cluster [DBG] pgmap v1247: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:34.701 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:40:34.701 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:40:35.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:34 smithi132 bash[18607]: cluster 2024-04-25T20:40:33.740822+0000 mgr.x (mgr.14254) 1530 : cluster [DBG] pgmap v1248: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:35.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:34 smithi132 bash[18607]: cluster 2024-04-25T20:40:33.740822+0000 mgr.x (mgr.14254) 1530 : cluster [DBG] pgmap v1248: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:35.359 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:40:36.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:35 smithi132 bash[18607]: audit 2024-04-25T20:40:34.699806+0000 mgr.x (mgr.14254) 1531 : audit [DBG] from='client.14984 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:40:36.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:35 smithi132 bash[18607]: audit 2024-04-25T20:40:34.699806+0000 mgr.x (mgr.14254) 1531 : audit [DBG] from='client.14984 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:40:36.361 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:40:37.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:36 smithi132 bash[18607]: cluster 2024-04-25T20:40:35.742235+0000 mgr.x (mgr.14254) 1532 : cluster [DBG] pgmap v1249: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:37.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:36 smithi132 bash[18607]: cluster 2024-04-25T20:40:35.742235+0000 mgr.x (mgr.14254) 1532 : cluster [DBG] pgmap v1249: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:39.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:38 smithi132 bash[18607]: cluster 2024-04-25T20:40:37.743646+0000 mgr.x (mgr.14254) 1533 : cluster [DBG] pgmap v1250: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:39.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:38 smithi132 bash[18607]: cluster 2024-04-25T20:40:37.743646+0000 mgr.x (mgr.14254) 1533 : cluster [DBG] pgmap v1250: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:41.121 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:40:41.142 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:40 smithi132 bash[18607]: cluster 2024-04-25T20:40:39.745545+0000 mgr.x (mgr.14254) 1534 : cluster [DBG] pgmap v1251: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:41.142 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:40 smithi132 bash[18607]: cluster 2024-04-25T20:40:39.745545+0000 mgr.x (mgr.14254) 1534 : cluster [DBG] pgmap v1251: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:43.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:42 smithi132 bash[18607]: cluster 2024-04-25T20:40:41.747020+0000 mgr.x (mgr.14254) 1535 : cluster [DBG] pgmap v1252: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:43.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:42 smithi132 bash[18607]: cluster 2024-04-25T20:40:41.747020+0000 mgr.x (mgr.14254) 1535 : cluster [DBG] pgmap v1252: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:43.985 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:40:43.985 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:40:44.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:43 smithi132 bash[18607]: cluster 2024-04-25T20:40:43.748127+0000 mgr.x (mgr.14254) 1536 : cluster [DBG] pgmap v1253: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:44.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:43 smithi132 bash[18607]: cluster 2024-04-25T20:40:43.748127+0000 mgr.x (mgr.14254) 1536 : cluster [DBG] pgmap v1253: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:44.745 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:40:45.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:44 smithi132 bash[18607]: audit 2024-04-25T20:40:43.986628+0000 mgr.x (mgr.14254) 1537 : audit [DBG] from='client.14986 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:40:45.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:44 smithi132 bash[18607]: audit 2024-04-25T20:40:43.986628+0000 mgr.x (mgr.14254) 1537 : audit [DBG] from='client.14986 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:40:45.746 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:40:46.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:45 smithi132 bash[18607]: cluster 2024-04-25T20:40:45.749680+0000 mgr.x (mgr.14254) 1538 : cluster [DBG] pgmap v1254: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:46.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:45 smithi132 bash[18607]: cluster 2024-04-25T20:40:45.749680+0000 mgr.x (mgr.14254) 1538 : cluster [DBG] pgmap v1254: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:49.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:48 smithi132 bash[18607]: cluster 2024-04-25T20:40:47.751138+0000 mgr.x (mgr.14254) 1539 : cluster [DBG] pgmap v1255: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:49.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:48 smithi132 bash[18607]: cluster 2024-04-25T20:40:47.751138+0000 mgr.x (mgr.14254) 1539 : cluster [DBG] pgmap v1255: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:50.509 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:40:51.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:50 smithi132 bash[18607]: cluster 2024-04-25T20:40:49.752989+0000 mgr.x (mgr.14254) 1540 : cluster [DBG] pgmap v1256: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:51.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:50 smithi132 bash[18607]: cluster 2024-04-25T20:40:49.752989+0000 mgr.x (mgr.14254) 1540 : cluster [DBG] pgmap v1256: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:53.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:52 smithi132 bash[18607]: cluster 2024-04-25T20:40:51.754363+0000 mgr.x (mgr.14254) 1541 : cluster [DBG] pgmap v1257: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:53.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:52 smithi132 bash[18607]: cluster 2024-04-25T20:40:51.754363+0000 mgr.x (mgr.14254) 1541 : cluster [DBG] pgmap v1257: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:53.418 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:40:53.419 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:40:54.121 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:40:55.122 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:40:55.133 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:54 smithi132 bash[18607]: audit 2024-04-25T20:40:53.417021+0000 mgr.x (mgr.14254) 1542 : audit [DBG] from='client.14988 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:40:55.133 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:54 smithi132 bash[18607]: audit 2024-04-25T20:40:53.417021+0000 mgr.x (mgr.14254) 1542 : audit [DBG] from='client.14988 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:40:55.134 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:54 smithi132 bash[18607]: cluster 2024-04-25T20:40:53.756231+0000 mgr.x (mgr.14254) 1543 : cluster [DBG] pgmap v1258: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:55.134 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:54 smithi132 bash[18607]: cluster 2024-04-25T20:40:53.756231+0000 mgr.x (mgr.14254) 1543 : cluster [DBG] pgmap v1258: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:57.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:56 smithi132 bash[18607]: cluster 2024-04-25T20:40:55.757669+0000 mgr.x (mgr.14254) 1544 : cluster [DBG] pgmap v1259: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:57.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:56 smithi132 bash[18607]: cluster 2024-04-25T20:40:55.757669+0000 mgr.x (mgr.14254) 1544 : cluster [DBG] pgmap v1259: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:59.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:58 smithi132 bash[18607]: cluster 2024-04-25T20:40:57.759149+0000 mgr.x (mgr.14254) 1545 : cluster [DBG] pgmap v1260: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:59.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:40:58 smithi132 bash[18607]: cluster 2024-04-25T20:40:57.759149+0000 mgr.x (mgr.14254) 1545 : cluster [DBG] pgmap v1260: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:40:59.881 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:41:00.934 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:00 smithi132 bash[18607]: cluster 2024-04-25T20:40:59.761048+0000 mgr.x (mgr.14254) 1546 : cluster [DBG] pgmap v1261: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:00.934 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:00 smithi132 bash[18607]: cluster 2024-04-25T20:40:59.761048+0000 mgr.x (mgr.14254) 1546 : cluster [DBG] pgmap v1261: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:02.612 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:41:02.613 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:41:03.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:02 smithi132 bash[18607]: cluster 2024-04-25T20:41:01.762412+0000 mgr.x (mgr.14254) 1547 : cluster [DBG] pgmap v1262: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:03.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:02 smithi132 bash[18607]: cluster 2024-04-25T20:41:01.762412+0000 mgr.x (mgr.14254) 1547 : cluster [DBG] pgmap v1262: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:03.345 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:41:04.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:03 smithi132 bash[18607]: audit 2024-04-25T20:41:02.611149+0000 mgr.x (mgr.14254) 1548 : audit [DBG] from='client.14990 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:41:04.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:03 smithi132 bash[18607]: audit 2024-04-25T20:41:02.611149+0000 mgr.x (mgr.14254) 1548 : audit [DBG] from='client.14990 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:41:04.346 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:41:05.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:04 smithi132 bash[18607]: cluster 2024-04-25T20:41:03.764214+0000 mgr.x (mgr.14254) 1549 : cluster [DBG] pgmap v1263: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:05.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:04 smithi132 bash[18607]: cluster 2024-04-25T20:41:03.764214+0000 mgr.x (mgr.14254) 1549 : cluster [DBG] pgmap v1263: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:07.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:06 smithi132 bash[18607]: cluster 2024-04-25T20:41:05.765628+0000 mgr.x (mgr.14254) 1550 : cluster [DBG] pgmap v1264: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:07.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:06 smithi132 bash[18607]: cluster 2024-04-25T20:41:05.765628+0000 mgr.x (mgr.14254) 1550 : cluster [DBG] pgmap v1264: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:09.099 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:41:09.120 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:08 smithi132 bash[18607]: cluster 2024-04-25T20:41:07.767016+0000 mgr.x (mgr.14254) 1551 : cluster [DBG] pgmap v1265: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:09.120 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:08 smithi132 bash[18607]: cluster 2024-04-25T20:41:07.767016+0000 mgr.x (mgr.14254) 1551 : cluster [DBG] pgmap v1265: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:11.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:10 smithi132 bash[18607]: cluster 2024-04-25T20:41:09.768735+0000 mgr.x (mgr.14254) 1552 : cluster [DBG] pgmap v1266: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:11.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:10 smithi132 bash[18607]: cluster 2024-04-25T20:41:09.768735+0000 mgr.x (mgr.14254) 1552 : cluster [DBG] pgmap v1266: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:11.916 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:41:11.917 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:41:12.603 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:41:13.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:12 smithi132 bash[18607]: cluster 2024-04-25T20:41:11.769623+0000 mgr.x (mgr.14254) 1553 : cluster [DBG] pgmap v1267: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:13.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:12 smithi132 bash[18607]: cluster 2024-04-25T20:41:11.769623+0000 mgr.x (mgr.14254) 1553 : cluster [DBG] pgmap v1267: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:13.604 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:41:14.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:13 smithi132 bash[18607]: audit 2024-04-25T20:41:11.918353+0000 mgr.x (mgr.14254) 1554 : audit [DBG] from='client.14992 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:41:14.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:13 smithi132 bash[18607]: audit 2024-04-25T20:41:11.918353+0000 mgr.x (mgr.14254) 1554 : audit [DBG] from='client.14992 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:41:15.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:14 smithi132 bash[18607]: cluster 2024-04-25T20:41:13.771359+0000 mgr.x (mgr.14254) 1555 : cluster [DBG] pgmap v1268: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:15.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:14 smithi132 bash[18607]: cluster 2024-04-25T20:41:13.771359+0000 mgr.x (mgr.14254) 1555 : cluster [DBG] pgmap v1268: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:17.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:16 smithi132 bash[18607]: cluster 2024-04-25T20:41:15.772775+0000 mgr.x (mgr.14254) 1556 : cluster [DBG] pgmap v1269: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:17.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:16 smithi132 bash[18607]: cluster 2024-04-25T20:41:15.772775+0000 mgr.x (mgr.14254) 1556 : cluster [DBG] pgmap v1269: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:17.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:16 smithi132 bash[18607]: audit 2024-04-25T20:41:16.383517+0000 mon.a (mon.0) 529 : audit [DBG] from='client.? 172.17.0.2:0/2379930467' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:41:17.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:16 smithi132 bash[18607]: audit 2024-04-25T20:41:16.383517+0000 mon.a (mon.0) 529 : audit [DBG] from='client.? 172.17.0.2:0/2379930467' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:41:18.366 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:41:19.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:18 smithi132 bash[18607]: cluster 2024-04-25T20:41:17.774114+0000 mgr.x (mgr.14254) 1557 : cluster [DBG] pgmap v1270: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:19.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:18 smithi132 bash[18607]: cluster 2024-04-25T20:41:17.774114+0000 mgr.x (mgr.14254) 1557 : cluster [DBG] pgmap v1270: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:21.253 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:41:21.253 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:41:21.266 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:20 smithi132 bash[18607]: cluster 2024-04-25T20:41:19.775951+0000 mgr.x (mgr.14254) 1558 : cluster [DBG] pgmap v1271: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:21.266 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:20 smithi132 bash[18607]: cluster 2024-04-25T20:41:19.775951+0000 mgr.x (mgr.14254) 1558 : cluster [DBG] pgmap v1271: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:22.026 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:41:23.028 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:41:23.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:22 smithi132 bash[18607]: audit 2024-04-25T20:41:21.252116+0000 mgr.x (mgr.14254) 1559 : audit [DBG] from='client.14996 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:41:23.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:22 smithi132 bash[18607]: audit 2024-04-25T20:41:21.252116+0000 mgr.x (mgr.14254) 1559 : audit [DBG] from='client.14996 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:41:23.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:22 smithi132 bash[18607]: cluster 2024-04-25T20:41:21.777437+0000 mgr.x (mgr.14254) 1560 : cluster [DBG] pgmap v1272: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:23.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:22 smithi132 bash[18607]: cluster 2024-04-25T20:41:21.777437+0000 mgr.x (mgr.14254) 1560 : cluster [DBG] pgmap v1272: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:25.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:24 smithi132 bash[18607]: cluster 2024-04-25T20:41:23.779323+0000 mgr.x (mgr.14254) 1561 : cluster [DBG] pgmap v1273: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:25.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:24 smithi132 bash[18607]: cluster 2024-04-25T20:41:23.779323+0000 mgr.x (mgr.14254) 1561 : cluster [DBG] pgmap v1273: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:27.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:26 smithi132 bash[18607]: cluster 2024-04-25T20:41:25.780034+0000 mgr.x (mgr.14254) 1562 : cluster [DBG] pgmap v1274: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:27.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:26 smithi132 bash[18607]: cluster 2024-04-25T20:41:25.780034+0000 mgr.x (mgr.14254) 1562 : cluster [DBG] pgmap v1274: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:27.787 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:41:29.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:28 smithi132 bash[18607]: cluster 2024-04-25T20:41:27.781559+0000 mgr.x (mgr.14254) 1563 : cluster [DBG] pgmap v1275: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:29.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:28 smithi132 bash[18607]: cluster 2024-04-25T20:41:27.781559+0000 mgr.x (mgr.14254) 1563 : cluster [DBG] pgmap v1275: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:30.740 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:41:30.740 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:41:31.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:30 smithi132 bash[18607]: cluster 2024-04-25T20:41:29.783447+0000 mgr.x (mgr.14254) 1564 : cluster [DBG] pgmap v1276: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:31.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:30 smithi132 bash[18607]: cluster 2024-04-25T20:41:29.783447+0000 mgr.x (mgr.14254) 1564 : cluster [DBG] pgmap v1276: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:31.423 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:41:32.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:31 smithi132 bash[18607]: audit 2024-04-25T20:41:30.738490+0000 mgr.x (mgr.14254) 1565 : audit [DBG] from='client.14998 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:41:32.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:31 smithi132 bash[18607]: audit 2024-04-25T20:41:30.738490+0000 mgr.x (mgr.14254) 1565 : audit [DBG] from='client.14998 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:41:32.424 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:41:33.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:32 smithi132 bash[18607]: cluster 2024-04-25T20:41:31.784919+0000 mgr.x (mgr.14254) 1566 : cluster [DBG] pgmap v1277: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:33.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:32 smithi132 bash[18607]: cluster 2024-04-25T20:41:31.784919+0000 mgr.x (mgr.14254) 1566 : cluster [DBG] pgmap v1277: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:35.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:34 smithi132 bash[18607]: cluster 2024-04-25T20:41:33.786719+0000 mgr.x (mgr.14254) 1567 : cluster [DBG] pgmap v1278: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:35.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:34 smithi132 bash[18607]: cluster 2024-04-25T20:41:33.786719+0000 mgr.x (mgr.14254) 1567 : cluster [DBG] pgmap v1278: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:37.183 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:41:37.205 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:36 smithi132 bash[18607]: cluster 2024-04-25T20:41:35.788057+0000 mgr.x (mgr.14254) 1568 : cluster [DBG] pgmap v1279: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:37.205 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:36 smithi132 bash[18607]: cluster 2024-04-25T20:41:35.788057+0000 mgr.x (mgr.14254) 1568 : cluster [DBG] pgmap v1279: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:39.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:38 smithi132 bash[18607]: cluster 2024-04-25T20:41:37.789429+0000 mgr.x (mgr.14254) 1569 : cluster [DBG] pgmap v1280: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:39.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:38 smithi132 bash[18607]: cluster 2024-04-25T20:41:37.789429+0000 mgr.x (mgr.14254) 1569 : cluster [DBG] pgmap v1280: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:40.048 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:41:40.049 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:41:40.775 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:41:41.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:40 smithi132 bash[18607]: cluster 2024-04-25T20:41:39.791220+0000 mgr.x (mgr.14254) 1570 : cluster [DBG] pgmap v1281: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:41.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:40 smithi132 bash[18607]: cluster 2024-04-25T20:41:39.791220+0000 mgr.x (mgr.14254) 1570 : cluster [DBG] pgmap v1281: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:41.777 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:41:42.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:41 smithi132 bash[18607]: audit 2024-04-25T20:41:40.046957+0000 mgr.x (mgr.14254) 1571 : audit [DBG] from='client.15000 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:41:42.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:41 smithi132 bash[18607]: audit 2024-04-25T20:41:40.046957+0000 mgr.x (mgr.14254) 1571 : audit [DBG] from='client.15000 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:41:43.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:42 smithi132 bash[18607]: cluster 2024-04-25T20:41:41.792636+0000 mgr.x (mgr.14254) 1572 : cluster [DBG] pgmap v1282: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:43.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:42 smithi132 bash[18607]: cluster 2024-04-25T20:41:41.792636+0000 mgr.x (mgr.14254) 1572 : cluster [DBG] pgmap v1282: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:45.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:44 smithi132 bash[18607]: cluster 2024-04-25T20:41:43.794793+0000 mgr.x (mgr.14254) 1573 : cluster [DBG] pgmap v1283: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:45.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:44 smithi132 bash[18607]: cluster 2024-04-25T20:41:43.794793+0000 mgr.x (mgr.14254) 1573 : cluster [DBG] pgmap v1283: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:46.540 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:41:47.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:46 smithi132 bash[18607]: cluster 2024-04-25T20:41:45.796237+0000 mgr.x (mgr.14254) 1574 : cluster [DBG] pgmap v1284: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:47.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:46 smithi132 bash[18607]: cluster 2024-04-25T20:41:45.796237+0000 mgr.x (mgr.14254) 1574 : cluster [DBG] pgmap v1284: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:49.154 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:48 smithi132 bash[18607]: cluster 2024-04-25T20:41:47.797153+0000 mgr.x (mgr.14254) 1575 : cluster [DBG] pgmap v1285: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:49.154 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:48 smithi132 bash[18607]: cluster 2024-04-25T20:41:47.797153+0000 mgr.x (mgr.14254) 1575 : cluster [DBG] pgmap v1285: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:49.522 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:41:49.523 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:41:50.200 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:41:51.201 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:41:51.212 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:50 smithi132 bash[18607]: audit 2024-04-25T20:41:49.521071+0000 mgr.x (mgr.14254) 1576 : audit [DBG] from='client.15002 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:41:51.212 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:50 smithi132 bash[18607]: audit 2024-04-25T20:41:49.521071+0000 mgr.x (mgr.14254) 1576 : audit [DBG] from='client.15002 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:41:51.212 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:50 smithi132 bash[18607]: cluster 2024-04-25T20:41:49.798956+0000 mgr.x (mgr.14254) 1577 : cluster [DBG] pgmap v1286: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:51.212 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:50 smithi132 bash[18607]: cluster 2024-04-25T20:41:49.798956+0000 mgr.x (mgr.14254) 1577 : cluster [DBG] pgmap v1286: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:53.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:52 smithi132 bash[18607]: cluster 2024-04-25T20:41:51.800360+0000 mgr.x (mgr.14254) 1578 : cluster [DBG] pgmap v1287: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:53.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:52 smithi132 bash[18607]: cluster 2024-04-25T20:41:51.800360+0000 mgr.x (mgr.14254) 1578 : cluster [DBG] pgmap v1287: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:55.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:54 smithi132 bash[18607]: cluster 2024-04-25T20:41:53.802248+0000 mgr.x (mgr.14254) 1579 : cluster [DBG] pgmap v1288: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:55.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:54 smithi132 bash[18607]: cluster 2024-04-25T20:41:53.802248+0000 mgr.x (mgr.14254) 1579 : cluster [DBG] pgmap v1288: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:55.963 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:41:57.168 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:56 smithi132 bash[18607]: cluster 2024-04-25T20:41:55.803659+0000 mgr.x (mgr.14254) 1580 : cluster [DBG] pgmap v1289: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:57.168 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:56 smithi132 bash[18607]: cluster 2024-04-25T20:41:55.803659+0000 mgr.x (mgr.14254) 1580 : cluster [DBG] pgmap v1289: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:58.881 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:41:58.881 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:41:59.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:58 smithi132 bash[18607]: cluster 2024-04-25T20:41:57.805066+0000 mgr.x (mgr.14254) 1581 : cluster [DBG] pgmap v1290: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:59.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:58 smithi132 bash[18607]: cluster 2024-04-25T20:41:57.805066+0000 mgr.x (mgr.14254) 1581 : cluster [DBG] pgmap v1290: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:41:59.608 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:42:00.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:59 smithi132 bash[18607]: audit 2024-04-25T20:41:58.882778+0000 mgr.x (mgr.14254) 1582 : audit [DBG] from='client.15004 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:42:00.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:41:59 smithi132 bash[18607]: audit 2024-04-25T20:41:58.882778+0000 mgr.x (mgr.14254) 1582 : audit [DBG] from='client.15004 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:42:00.609 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:42:01.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:00 smithi132 bash[18607]: cluster 2024-04-25T20:41:59.807109+0000 mgr.x (mgr.14254) 1583 : cluster [DBG] pgmap v1291: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:01.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:00 smithi132 bash[18607]: cluster 2024-04-25T20:41:59.807109+0000 mgr.x (mgr.14254) 1583 : cluster [DBG] pgmap v1291: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:03.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:02 smithi132 bash[18607]: cluster 2024-04-25T20:42:01.808789+0000 mgr.x (mgr.14254) 1584 : cluster [DBG] pgmap v1292: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:03.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:02 smithi132 bash[18607]: cluster 2024-04-25T20:42:01.808789+0000 mgr.x (mgr.14254) 1584 : cluster [DBG] pgmap v1292: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:05.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:04 smithi132 bash[18607]: cluster 2024-04-25T20:42:03.810723+0000 mgr.x (mgr.14254) 1585 : cluster [DBG] pgmap v1293: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:05.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:04 smithi132 bash[18607]: cluster 2024-04-25T20:42:03.810723+0000 mgr.x (mgr.14254) 1585 : cluster [DBG] pgmap v1293: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:05.377 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:42:07.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:06 smithi132 bash[18607]: cluster 2024-04-25T20:42:05.812324+0000 mgr.x (mgr.14254) 1586 : cluster [DBG] pgmap v1294: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:07.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:06 smithi132 bash[18607]: cluster 2024-04-25T20:42:05.812324+0000 mgr.x (mgr.14254) 1586 : cluster [DBG] pgmap v1294: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:08.245 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:42:08.245 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:42:08.927 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:42:09.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:08 smithi132 bash[18607]: cluster 2024-04-25T20:42:07.813876+0000 mgr.x (mgr.14254) 1587 : cluster [DBG] pgmap v1295: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:09.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:08 smithi132 bash[18607]: cluster 2024-04-25T20:42:07.813876+0000 mgr.x (mgr.14254) 1587 : cluster [DBG] pgmap v1295: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:09.929 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:42:10.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:09 smithi132 bash[18607]: audit 2024-04-25T20:42:08.245911+0000 mgr.x (mgr.14254) 1588 : audit [DBG] from='client.15006 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:42:10.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:09 smithi132 bash[18607]: audit 2024-04-25T20:42:08.245911+0000 mgr.x (mgr.14254) 1588 : audit [DBG] from='client.15006 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:42:11.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:10 smithi132 bash[18607]: cluster 2024-04-25T20:42:09.815734+0000 mgr.x (mgr.14254) 1589 : cluster [DBG] pgmap v1296: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:11.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:10 smithi132 bash[18607]: cluster 2024-04-25T20:42:09.815734+0000 mgr.x (mgr.14254) 1589 : cluster [DBG] pgmap v1296: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:12.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:11 smithi132 bash[18607]: cluster 2024-04-25T20:42:11.817159+0000 mgr.x (mgr.14254) 1590 : cluster [DBG] pgmap v1297: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:12.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:11 smithi132 bash[18607]: cluster 2024-04-25T20:42:11.817159+0000 mgr.x (mgr.14254) 1590 : cluster [DBG] pgmap v1297: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:14.692 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:42:15.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:14 smithi132 bash[18607]: cluster 2024-04-25T20:42:13.818989+0000 mgr.x (mgr.14254) 1591 : cluster [DBG] pgmap v1298: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:15.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:14 smithi132 bash[18607]: cluster 2024-04-25T20:42:13.818989+0000 mgr.x (mgr.14254) 1591 : cluster [DBG] pgmap v1298: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:17.125 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:16 smithi132 bash[18607]: cluster 2024-04-25T20:42:15.820006+0000 mgr.x (mgr.14254) 1592 : cluster [DBG] pgmap v1299: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:17.126 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:16 smithi132 bash[18607]: cluster 2024-04-25T20:42:15.820006+0000 mgr.x (mgr.14254) 1592 : cluster [DBG] pgmap v1299: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:17.495 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:42:17.495 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:42:18.170 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:42:19.171 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:42:19.182 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:18 smithi132 bash[18607]: audit 2024-04-25T20:42:17.496392+0000 mgr.x (mgr.14254) 1593 : audit [DBG] from='client.15008 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:42:19.182 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:18 smithi132 bash[18607]: audit 2024-04-25T20:42:17.496392+0000 mgr.x (mgr.14254) 1593 : audit [DBG] from='client.15008 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:42:19.182 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:18 smithi132 bash[18607]: cluster 2024-04-25T20:42:17.821442+0000 mgr.x (mgr.14254) 1594 : cluster [DBG] pgmap v1300: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:19.182 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:18 smithi132 bash[18607]: cluster 2024-04-25T20:42:17.821442+0000 mgr.x (mgr.14254) 1594 : cluster [DBG] pgmap v1300: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:21.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:20 smithi132 bash[18607]: cluster 2024-04-25T20:42:19.823293+0000 mgr.x (mgr.14254) 1595 : cluster [DBG] pgmap v1301: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:21.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:20 smithi132 bash[18607]: cluster 2024-04-25T20:42:19.823293+0000 mgr.x (mgr.14254) 1595 : cluster [DBG] pgmap v1301: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:23.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:22 smithi132 bash[18607]: cluster 2024-04-25T20:42:21.824714+0000 mgr.x (mgr.14254) 1596 : cluster [DBG] pgmap v1302: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:23.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:22 smithi132 bash[18607]: cluster 2024-04-25T20:42:21.824714+0000 mgr.x (mgr.14254) 1596 : cluster [DBG] pgmap v1302: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:23.937 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:42:25.171 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:24 smithi132 bash[18607]: cluster 2024-04-25T20:42:23.826692+0000 mgr.x (mgr.14254) 1597 : cluster [DBG] pgmap v1303: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:25.171 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:24 smithi132 bash[18607]: cluster 2024-04-25T20:42:23.826692+0000 mgr.x (mgr.14254) 1597 : cluster [DBG] pgmap v1303: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:26.846 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:42:26.846 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:42:27.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:26 smithi132 bash[18607]: cluster 2024-04-25T20:42:25.828354+0000 mgr.x (mgr.14254) 1598 : cluster [DBG] pgmap v1304: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:27.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:26 smithi132 bash[18607]: cluster 2024-04-25T20:42:25.828354+0000 mgr.x (mgr.14254) 1598 : cluster [DBG] pgmap v1304: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:27.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:26 smithi132 bash[18607]: audit 2024-04-25T20:42:26.772103+0000 mon.a (mon.0) 530 : audit [DBG] from='client.? 172.17.0.2:0/2701527907' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:42:27.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:26 smithi132 bash[18607]: audit 2024-04-25T20:42:26.772103+0000 mon.a (mon.0) 530 : audit [DBG] from='client.? 172.17.0.2:0/2701527907' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:42:27.630 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:42:28.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:27 smithi132 bash[18607]: audit 2024-04-25T20:42:26.847572+0000 mgr.x (mgr.14254) 1599 : audit [DBG] from='client.15010 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:42:28.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:27 smithi132 bash[18607]: audit 2024-04-25T20:42:26.847572+0000 mgr.x (mgr.14254) 1599 : audit [DBG] from='client.15010 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:42:28.631 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:42:29.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:28 smithi132 bash[18607]: cluster 2024-04-25T20:42:27.829881+0000 mgr.x (mgr.14254) 1600 : cluster [DBG] pgmap v1305: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:29.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:28 smithi132 bash[18607]: cluster 2024-04-25T20:42:27.829881+0000 mgr.x (mgr.14254) 1600 : cluster [DBG] pgmap v1305: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:31.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:30 smithi132 bash[18607]: cluster 2024-04-25T20:42:29.831780+0000 mgr.x (mgr.14254) 1601 : cluster [DBG] pgmap v1306: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:31.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:30 smithi132 bash[18607]: cluster 2024-04-25T20:42:29.831780+0000 mgr.x (mgr.14254) 1601 : cluster [DBG] pgmap v1306: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:33.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:32 smithi132 bash[18607]: cluster 2024-04-25T20:42:31.833297+0000 mgr.x (mgr.14254) 1602 : cluster [DBG] pgmap v1307: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:33.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:32 smithi132 bash[18607]: cluster 2024-04-25T20:42:31.833297+0000 mgr.x (mgr.14254) 1602 : cluster [DBG] pgmap v1307: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:33.383 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:42:35.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:34 smithi132 bash[18607]: cluster 2024-04-25T20:42:33.835181+0000 mgr.x (mgr.14254) 1603 : cluster [DBG] pgmap v1308: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:35.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:34 smithi132 bash[18607]: cluster 2024-04-25T20:42:33.835181+0000 mgr.x (mgr.14254) 1603 : cluster [DBG] pgmap v1308: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:36.087 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:42:36.088 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:42:36.775 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:42:37.074 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:36 smithi132 bash[18607]: cluster 2024-04-25T20:42:35.836624+0000 mgr.x (mgr.14254) 1604 : cluster [DBG] pgmap v1309: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:37.074 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:36 smithi132 bash[18607]: cluster 2024-04-25T20:42:35.836624+0000 mgr.x (mgr.14254) 1604 : cluster [DBG] pgmap v1309: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:37.776 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:42:38.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:37 smithi132 bash[18607]: audit 2024-04-25T20:42:36.088397+0000 mgr.x (mgr.14254) 1605 : audit [DBG] from='client.15014 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:42:38.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:37 smithi132 bash[18607]: audit 2024-04-25T20:42:36.088397+0000 mgr.x (mgr.14254) 1605 : audit [DBG] from='client.15014 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:42:39.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:38 smithi132 bash[18607]: cluster 2024-04-25T20:42:37.837601+0000 mgr.x (mgr.14254) 1606 : cluster [DBG] pgmap v1310: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:39.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:38 smithi132 bash[18607]: cluster 2024-04-25T20:42:37.837601+0000 mgr.x (mgr.14254) 1606 : cluster [DBG] pgmap v1310: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:41.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:40 smithi132 bash[18607]: cluster 2024-04-25T20:42:39.839424+0000 mgr.x (mgr.14254) 1607 : cluster [DBG] pgmap v1311: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:41.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:40 smithi132 bash[18607]: cluster 2024-04-25T20:42:39.839424+0000 mgr.x (mgr.14254) 1607 : cluster [DBG] pgmap v1311: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:42.536 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:42:43.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:42 smithi132 bash[18607]: cluster 2024-04-25T20:42:41.840974+0000 mgr.x (mgr.14254) 1608 : cluster [DBG] pgmap v1312: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:43.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:42 smithi132 bash[18607]: cluster 2024-04-25T20:42:41.840974+0000 mgr.x (mgr.14254) 1608 : cluster [DBG] pgmap v1312: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:45.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:44 smithi132 bash[18607]: cluster 2024-04-25T20:42:43.842829+0000 mgr.x (mgr.14254) 1609 : cluster [DBG] pgmap v1313: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:45.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:44 smithi132 bash[18607]: cluster 2024-04-25T20:42:43.842829+0000 mgr.x (mgr.14254) 1609 : cluster [DBG] pgmap v1313: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:45.461 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:42:45.461 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:42:46.163 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:42:47.164 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:42:47.175 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:46 smithi132 bash[18607]: audit 2024-04-25T20:42:45.460250+0000 mgr.x (mgr.14254) 1610 : audit [DBG] from='client.15016 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:42:47.176 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:46 smithi132 bash[18607]: audit 2024-04-25T20:42:45.460250+0000 mgr.x (mgr.14254) 1610 : audit [DBG] from='client.15016 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:42:47.176 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:46 smithi132 bash[18607]: cluster 2024-04-25T20:42:45.844387+0000 mgr.x (mgr.14254) 1611 : cluster [DBG] pgmap v1314: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:47.176 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:46 smithi132 bash[18607]: cluster 2024-04-25T20:42:45.844387+0000 mgr.x (mgr.14254) 1611 : cluster [DBG] pgmap v1314: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:49.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:48 smithi132 bash[18607]: cluster 2024-04-25T20:42:47.846015+0000 mgr.x (mgr.14254) 1612 : cluster [DBG] pgmap v1315: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:49.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:48 smithi132 bash[18607]: cluster 2024-04-25T20:42:47.846015+0000 mgr.x (mgr.14254) 1612 : cluster [DBG] pgmap v1315: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:51.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:50 smithi132 bash[18607]: cluster 2024-04-25T20:42:49.848038+0000 mgr.x (mgr.14254) 1613 : cluster [DBG] pgmap v1316: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:51.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:50 smithi132 bash[18607]: cluster 2024-04-25T20:42:49.848038+0000 mgr.x (mgr.14254) 1613 : cluster [DBG] pgmap v1316: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:51.926 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:42:52.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:51 smithi132 bash[18607]: cluster 2024-04-25T20:42:51.849492+0000 mgr.x (mgr.14254) 1614 : cluster [DBG] pgmap v1317: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:52.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:51 smithi132 bash[18607]: cluster 2024-04-25T20:42:51.849492+0000 mgr.x (mgr.14254) 1614 : cluster [DBG] pgmap v1317: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:54.849 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:42:54.850 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:42:55.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:54 smithi132 bash[18607]: cluster 2024-04-25T20:42:53.851473+0000 mgr.x (mgr.14254) 1615 : cluster [DBG] pgmap v1318: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:55.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:54 smithi132 bash[18607]: cluster 2024-04-25T20:42:53.851473+0000 mgr.x (mgr.14254) 1615 : cluster [DBG] pgmap v1318: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:55.654 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:42:56.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:55 smithi132 bash[18607]: audit 2024-04-25T20:42:54.848868+0000 mgr.x (mgr.14254) 1616 : audit [DBG] from='client.15018 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:42:56.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:55 smithi132 bash[18607]: audit 2024-04-25T20:42:54.848868+0000 mgr.x (mgr.14254) 1616 : audit [DBG] from='client.15018 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:42:56.718 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:42:57.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:56 smithi132 bash[18607]: cluster 2024-04-25T20:42:55.853093+0000 mgr.x (mgr.14254) 1617 : cluster [DBG] pgmap v1319: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:57.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:56 smithi132 bash[18607]: cluster 2024-04-25T20:42:55.853093+0000 mgr.x (mgr.14254) 1617 : cluster [DBG] pgmap v1319: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:59.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:58 smithi132 bash[18607]: cluster 2024-04-25T20:42:57.854644+0000 mgr.x (mgr.14254) 1618 : cluster [DBG] pgmap v1320: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:42:59.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:42:58 smithi132 bash[18607]: cluster 2024-04-25T20:42:57.854644+0000 mgr.x (mgr.14254) 1618 : cluster [DBG] pgmap v1320: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:01.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:00 smithi132 bash[18607]: cluster 2024-04-25T20:42:59.856637+0000 mgr.x (mgr.14254) 1619 : cluster [DBG] pgmap v1321: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:01.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:00 smithi132 bash[18607]: cluster 2024-04-25T20:42:59.856637+0000 mgr.x (mgr.14254) 1619 : cluster [DBG] pgmap v1321: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:01.468 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:43:03.195 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:02 smithi132 bash[18607]: cluster 2024-04-25T20:43:01.858103+0000 mgr.x (mgr.14254) 1620 : cluster [DBG] pgmap v1322: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:03.195 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:02 smithi132 bash[18607]: cluster 2024-04-25T20:43:01.858103+0000 mgr.x (mgr.14254) 1620 : cluster [DBG] pgmap v1322: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:04.376 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:43:04.376 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:43:05.052 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:43:05.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:04 smithi132 bash[18607]: cluster 2024-04-25T20:43:03.860068+0000 mgr.x (mgr.14254) 1621 : cluster [DBG] pgmap v1323: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:05.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:04 smithi132 bash[18607]: cluster 2024-04-25T20:43:03.860068+0000 mgr.x (mgr.14254) 1621 : cluster [DBG] pgmap v1323: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:06.054 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:43:06.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:05 smithi132 bash[18607]: audit 2024-04-25T20:43:04.375152+0000 mgr.x (mgr.14254) 1622 : audit [DBG] from='client.15020 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:43:06.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:05 smithi132 bash[18607]: audit 2024-04-25T20:43:04.375152+0000 mgr.x (mgr.14254) 1622 : audit [DBG] from='client.15020 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:43:07.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:06 smithi132 bash[18607]: cluster 2024-04-25T20:43:05.861583+0000 mgr.x (mgr.14254) 1623 : cluster [DBG] pgmap v1324: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:07.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:06 smithi132 bash[18607]: cluster 2024-04-25T20:43:05.861583+0000 mgr.x (mgr.14254) 1623 : cluster [DBG] pgmap v1324: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:08.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:07 smithi132 bash[18607]: cluster 2024-04-25T20:43:07.862983+0000 mgr.x (mgr.14254) 1624 : cluster [DBG] pgmap v1325: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:08.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:07 smithi132 bash[18607]: cluster 2024-04-25T20:43:07.862983+0000 mgr.x (mgr.14254) 1624 : cluster [DBG] pgmap v1325: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:10.814 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:43:11.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:10 smithi132 bash[18607]: cluster 2024-04-25T20:43:09.864852+0000 mgr.x (mgr.14254) 1625 : cluster [DBG] pgmap v1326: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:11.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:10 smithi132 bash[18607]: cluster 2024-04-25T20:43:09.864852+0000 mgr.x (mgr.14254) 1625 : cluster [DBG] pgmap v1326: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:13.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:12 smithi132 bash[18607]: cluster 2024-04-25T20:43:11.865660+0000 mgr.x (mgr.14254) 1626 : cluster [DBG] pgmap v1327: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:13.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:12 smithi132 bash[18607]: cluster 2024-04-25T20:43:11.865660+0000 mgr.x (mgr.14254) 1626 : cluster [DBG] pgmap v1327: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:13.776 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:43:13.776 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:43:14.489 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:43:15.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:14 smithi132 bash[18607]: audit 2024-04-25T20:43:13.774963+0000 mgr.x (mgr.14254) 1627 : audit [DBG] from='client.15022 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:43:15.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:14 smithi132 bash[18607]: audit 2024-04-25T20:43:13.774963+0000 mgr.x (mgr.14254) 1627 : audit [DBG] from='client.15022 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:43:15.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:14 smithi132 bash[18607]: cluster 2024-04-25T20:43:13.866504+0000 mgr.x (mgr.14254) 1628 : cluster [DBG] pgmap v1328: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:15.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:14 smithi132 bash[18607]: cluster 2024-04-25T20:43:13.866504+0000 mgr.x (mgr.14254) 1628 : cluster [DBG] pgmap v1328: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:15.490 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:43:17.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:16 smithi132 bash[18607]: cluster 2024-04-25T20:43:15.867984+0000 mgr.x (mgr.14254) 1629 : cluster [DBG] pgmap v1329: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:17.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:16 smithi132 bash[18607]: cluster 2024-04-25T20:43:15.867984+0000 mgr.x (mgr.14254) 1629 : cluster [DBG] pgmap v1329: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:19.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:18 smithi132 bash[18607]: cluster 2024-04-25T20:43:17.869553+0000 mgr.x (mgr.14254) 1630 : cluster [DBG] pgmap v1330: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:19.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:18 smithi132 bash[18607]: cluster 2024-04-25T20:43:17.869553+0000 mgr.x (mgr.14254) 1630 : cluster [DBG] pgmap v1330: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:20.252 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:43:20.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:19 smithi132 bash[18607]: cluster 2024-04-25T20:43:19.871553+0000 mgr.x (mgr.14254) 1631 : cluster [DBG] pgmap v1331: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:20.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:19 smithi132 bash[18607]: cluster 2024-04-25T20:43:19.871553+0000 mgr.x (mgr.14254) 1631 : cluster [DBG] pgmap v1331: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:22.948 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:43:22.948 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:43:23.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:22 smithi132 bash[18607]: cluster 2024-04-25T20:43:21.872596+0000 mgr.x (mgr.14254) 1632 : cluster [DBG] pgmap v1332: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:23.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:22 smithi132 bash[18607]: cluster 2024-04-25T20:43:21.872596+0000 mgr.x (mgr.14254) 1632 : cluster [DBG] pgmap v1332: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:23.759 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:43:24.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:23 smithi132 bash[18607]: audit 2024-04-25T20:43:22.946628+0000 mgr.x (mgr.14254) 1633 : audit [DBG] from='client.15024 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:43:24.020 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:23 smithi132 bash[18607]: audit 2024-04-25T20:43:22.946628+0000 mgr.x (mgr.14254) 1633 : audit [DBG] from='client.15024 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:43:24.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:23 smithi132 bash[18607]: cluster 2024-04-25T20:43:23.874473+0000 mgr.x (mgr.14254) 1634 : cluster [DBG] pgmap v1333: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:24.021 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:23 smithi132 bash[18607]: cluster 2024-04-25T20:43:23.874473+0000 mgr.x (mgr.14254) 1634 : cluster [DBG] pgmap v1333: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:24.760 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:43:26.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:25 smithi132 bash[18607]: cluster 2024-04-25T20:43:25.875873+0000 mgr.x (mgr.14254) 1635 : cluster [DBG] pgmap v1334: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:26.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:25 smithi132 bash[18607]: cluster 2024-04-25T20:43:25.875873+0000 mgr.x (mgr.14254) 1635 : cluster [DBG] pgmap v1334: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:28.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:27 smithi132 bash[18607]: cluster 2024-04-25T20:43:27.877345+0000 mgr.x (mgr.14254) 1636 : cluster [DBG] pgmap v1335: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:28.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:27 smithi132 bash[18607]: cluster 2024-04-25T20:43:27.877345+0000 mgr.x (mgr.14254) 1636 : cluster [DBG] pgmap v1335: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:29.515 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:43:30.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:29 smithi132 bash[18607]: cluster 2024-04-25T20:43:29.878256+0000 mgr.x (mgr.14254) 1637 : cluster [DBG] pgmap v1336: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:30.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:29 smithi132 bash[18607]: cluster 2024-04-25T20:43:29.878256+0000 mgr.x (mgr.14254) 1637 : cluster [DBG] pgmap v1336: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:32.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:31 smithi132 bash[18607]: cluster 2024-04-25T20:43:31.879699+0000 mgr.x (mgr.14254) 1638 : cluster [DBG] pgmap v1337: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:32.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:31 smithi132 bash[18607]: cluster 2024-04-25T20:43:31.879699+0000 mgr.x (mgr.14254) 1638 : cluster [DBG] pgmap v1337: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:32.372 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:43:32.372 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:43:33.091 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:43:33.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:32 smithi132 bash[18607]: audit 2024-04-25T20:43:32.371018+0000 mgr.x (mgr.14254) 1639 : audit [DBG] from='client.15026 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:43:33.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:32 smithi132 bash[18607]: audit 2024-04-25T20:43:32.371018+0000 mgr.x (mgr.14254) 1639 : audit [DBG] from='client.15026 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:43:34.092 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:43:35.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:34 smithi132 bash[18607]: cluster 2024-04-25T20:43:33.881725+0000 mgr.x (mgr.14254) 1640 : cluster [DBG] pgmap v1338: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:35.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:34 smithi132 bash[18607]: cluster 2024-04-25T20:43:33.881725+0000 mgr.x (mgr.14254) 1640 : cluster [DBG] pgmap v1338: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:36.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:35 smithi132 bash[18607]: cluster 2024-04-25T20:43:35.883292+0000 mgr.x (mgr.14254) 1641 : cluster [DBG] pgmap v1339: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:36.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:35 smithi132 bash[18607]: cluster 2024-04-25T20:43:35.883292+0000 mgr.x (mgr.14254) 1641 : cluster [DBG] pgmap v1339: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:37.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:37 smithi132 bash[18607]: audit 2024-04-25T20:43:37.159666+0000 mon.a (mon.0) 531 : audit [DBG] from='client.? 172.17.0.2:0/1775184365' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:43:37.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:37 smithi132 bash[18607]: audit 2024-04-25T20:43:37.159666+0000 mon.a (mon.0) 531 : audit [DBG] from='client.? 172.17.0.2:0/1775184365' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:43:38.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:38 smithi132 bash[18607]: cluster 2024-04-25T20:43:37.884822+0000 mgr.x (mgr.14254) 1642 : cluster [DBG] pgmap v1340: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:38.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:38 smithi132 bash[18607]: cluster 2024-04-25T20:43:37.884822+0000 mgr.x (mgr.14254) 1642 : cluster [DBG] pgmap v1340: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:38.852 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:43:40.167 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:39 smithi132 bash[18607]: cluster 2024-04-25T20:43:39.886106+0000 mgr.x (mgr.14254) 1643 : cluster [DBG] pgmap v1341: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:40.167 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:39 smithi132 bash[18607]: cluster 2024-04-25T20:43:39.886106+0000 mgr.x (mgr.14254) 1643 : cluster [DBG] pgmap v1341: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:41.778 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:43:41.778 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:43:42.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:41 smithi132 bash[18607]: audit 2024-04-25T20:43:41.779971+0000 mgr.x (mgr.14254) 1644 : audit [DBG] from='client.15030 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:43:42.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:41 smithi132 bash[18607]: audit 2024-04-25T20:43:41.779971+0000 mgr.x (mgr.14254) 1644 : audit [DBG] from='client.15030 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:43:42.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:41 smithi132 bash[18607]: cluster 2024-04-25T20:43:41.887469+0000 mgr.x (mgr.14254) 1645 : cluster [DBG] pgmap v1342: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:42.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:41 smithi132 bash[18607]: cluster 2024-04-25T20:43:41.887469+0000 mgr.x (mgr.14254) 1645 : cluster [DBG] pgmap v1342: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:42.521 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:43:43.523 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:43:44.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:43 smithi132 bash[18607]: cluster 2024-04-25T20:43:43.889341+0000 mgr.x (mgr.14254) 1646 : cluster [DBG] pgmap v1343: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:44.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:43 smithi132 bash[18607]: cluster 2024-04-25T20:43:43.889341+0000 mgr.x (mgr.14254) 1646 : cluster [DBG] pgmap v1343: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:46.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:45 smithi132 bash[18607]: cluster 2024-04-25T20:43:45.890802+0000 mgr.x (mgr.14254) 1647 : cluster [DBG] pgmap v1344: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:46.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:45 smithi132 bash[18607]: cluster 2024-04-25T20:43:45.890802+0000 mgr.x (mgr.14254) 1647 : cluster [DBG] pgmap v1344: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:48.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:47 smithi132 bash[18607]: cluster 2024-04-25T20:43:47.892194+0000 mgr.x (mgr.14254) 1648 : cluster [DBG] pgmap v1345: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:48.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:47 smithi132 bash[18607]: cluster 2024-04-25T20:43:47.892194+0000 mgr.x (mgr.14254) 1648 : cluster [DBG] pgmap v1345: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:48.283 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:43:50.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:49 smithi132 bash[18607]: cluster 2024-04-25T20:43:49.894084+0000 mgr.x (mgr.14254) 1649 : cluster [DBG] pgmap v1346: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:50.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:49 smithi132 bash[18607]: cluster 2024-04-25T20:43:49.894084+0000 mgr.x (mgr.14254) 1649 : cluster [DBG] pgmap v1346: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:51.207 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:43:51.208 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:43:51.926 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:43:52.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:51 smithi132 bash[18607]: audit 2024-04-25T20:43:51.208537+0000 mgr.x (mgr.14254) 1650 : audit [DBG] from='client.15032 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:43:52.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:51 smithi132 bash[18607]: audit 2024-04-25T20:43:51.208537+0000 mgr.x (mgr.14254) 1650 : audit [DBG] from='client.15032 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:43:52.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:51 smithi132 bash[18607]: cluster 2024-04-25T20:43:51.895538+0000 mgr.x (mgr.14254) 1651 : cluster [DBG] pgmap v1347: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:52.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:51 smithi132 bash[18607]: cluster 2024-04-25T20:43:51.895538+0000 mgr.x (mgr.14254) 1651 : cluster [DBG] pgmap v1347: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:52.927 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:43:54.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:53 smithi132 bash[18607]: cluster 2024-04-25T20:43:53.897449+0000 mgr.x (mgr.14254) 1652 : cluster [DBG] pgmap v1348: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:54.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:53 smithi132 bash[18607]: cluster 2024-04-25T20:43:53.897449+0000 mgr.x (mgr.14254) 1652 : cluster [DBG] pgmap v1348: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:56.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:55 smithi132 bash[18607]: cluster 2024-04-25T20:43:55.898869+0000 mgr.x (mgr.14254) 1653 : cluster [DBG] pgmap v1349: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:56.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:55 smithi132 bash[18607]: cluster 2024-04-25T20:43:55.898869+0000 mgr.x (mgr.14254) 1653 : cluster [DBG] pgmap v1349: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:57.688 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:43:58.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:57 smithi132 bash[18607]: cluster 2024-04-25T20:43:57.900302+0000 mgr.x (mgr.14254) 1654 : cluster [DBG] pgmap v1350: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:43:58.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:57 smithi132 bash[18607]: cluster 2024-04-25T20:43:57.900302+0000 mgr.x (mgr.14254) 1654 : cluster [DBG] pgmap v1350: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:00.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:59 smithi132 bash[18607]: cluster 2024-04-25T20:43:59.901481+0000 mgr.x (mgr.14254) 1655 : cluster [DBG] pgmap v1351: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:00.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:43:59 smithi132 bash[18607]: cluster 2024-04-25T20:43:59.901481+0000 mgr.x (mgr.14254) 1655 : cluster [DBG] pgmap v1351: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:00.506 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:44:00.506 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:44:01.251 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:00 smithi132 bash[18607]: audit 2024-04-25T20:44:00.504500+0000 mgr.x (mgr.14254) 1656 : audit [DBG] from='client.15034 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:44:01.252 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:00 smithi132 bash[18607]: audit 2024-04-25T20:44:00.504500+0000 mgr.x (mgr.14254) 1656 : audit [DBG] from='client.15034 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:44:01.253 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:44:02.255 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:44:02.265 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:01 smithi132 bash[18607]: cluster 2024-04-25T20:44:01.902875+0000 mgr.x (mgr.14254) 1657 : cluster [DBG] pgmap v1352: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:02.265 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:01 smithi132 bash[18607]: cluster 2024-04-25T20:44:01.902875+0000 mgr.x (mgr.14254) 1657 : cluster [DBG] pgmap v1352: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:04.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:03 smithi132 bash[18607]: cluster 2024-04-25T20:44:03.904800+0000 mgr.x (mgr.14254) 1658 : cluster [DBG] pgmap v1353: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:04.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:03 smithi132 bash[18607]: cluster 2024-04-25T20:44:03.904800+0000 mgr.x (mgr.14254) 1658 : cluster [DBG] pgmap v1353: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:06.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:05 smithi132 bash[18607]: cluster 2024-04-25T20:44:05.906219+0000 mgr.x (mgr.14254) 1659 : cluster [DBG] pgmap v1354: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:06.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:05 smithi132 bash[18607]: cluster 2024-04-25T20:44:05.906219+0000 mgr.x (mgr.14254) 1659 : cluster [DBG] pgmap v1354: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:07.017 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:44:08.222 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:07 smithi132 bash[18607]: cluster 2024-04-25T20:44:07.907379+0000 mgr.x (mgr.14254) 1660 : cluster [DBG] pgmap v1355: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:08.222 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:07 smithi132 bash[18607]: cluster 2024-04-25T20:44:07.907379+0000 mgr.x (mgr.14254) 1660 : cluster [DBG] pgmap v1355: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:09.857 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:44:09.857 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:44:10.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:09 smithi132 bash[18607]: audit 2024-04-25T20:44:09.858095+0000 mgr.x (mgr.14254) 1661 : audit [DBG] from='client.15036 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:44:10.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:09 smithi132 bash[18607]: audit 2024-04-25T20:44:09.858095+0000 mgr.x (mgr.14254) 1661 : audit [DBG] from='client.15036 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:44:10.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:09 smithi132 bash[18607]: cluster 2024-04-25T20:44:09.908278+0000 mgr.x (mgr.14254) 1662 : cluster [DBG] pgmap v1356: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:10.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:09 smithi132 bash[18607]: cluster 2024-04-25T20:44:09.908278+0000 mgr.x (mgr.14254) 1662 : cluster [DBG] pgmap v1356: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:10.598 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:44:11.599 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:44:12.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:11 smithi132 bash[18607]: cluster 2024-04-25T20:44:11.909846+0000 mgr.x (mgr.14254) 1663 : cluster [DBG] pgmap v1357: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:12.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:11 smithi132 bash[18607]: cluster 2024-04-25T20:44:11.909846+0000 mgr.x (mgr.14254) 1663 : cluster [DBG] pgmap v1357: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:14.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:13 smithi132 bash[18607]: cluster 2024-04-25T20:44:13.911699+0000 mgr.x (mgr.14254) 1664 : cluster [DBG] pgmap v1358: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:14.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:13 smithi132 bash[18607]: cluster 2024-04-25T20:44:13.911699+0000 mgr.x (mgr.14254) 1664 : cluster [DBG] pgmap v1358: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:16.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:15 smithi132 bash[18607]: cluster 2024-04-25T20:44:15.913176+0000 mgr.x (mgr.14254) 1665 : cluster [DBG] pgmap v1359: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:16.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:15 smithi132 bash[18607]: cluster 2024-04-25T20:44:15.913176+0000 mgr.x (mgr.14254) 1665 : cluster [DBG] pgmap v1359: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:16.357 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:44:18.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:17 smithi132 bash[18607]: cluster 2024-04-25T20:44:17.914998+0000 mgr.x (mgr.14254) 1666 : cluster [DBG] pgmap v1360: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:18.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:17 smithi132 bash[18607]: cluster 2024-04-25T20:44:17.914998+0000 mgr.x (mgr.14254) 1666 : cluster [DBG] pgmap v1360: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:19.355 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:44:19.356 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:44:20.023 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:44:20.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:19 smithi132 bash[18607]: audit 2024-04-25T20:44:19.356741+0000 mgr.x (mgr.14254) 1667 : audit [DBG] from='client.15038 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:44:20.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:19 smithi132 bash[18607]: audit 2024-04-25T20:44:19.356741+0000 mgr.x (mgr.14254) 1667 : audit [DBG] from='client.15038 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:44:20.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:19 smithi132 bash[18607]: cluster 2024-04-25T20:44:19.916583+0000 mgr.x (mgr.14254) 1668 : cluster [DBG] pgmap v1361: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:20.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:19 smithi132 bash[18607]: cluster 2024-04-25T20:44:19.916583+0000 mgr.x (mgr.14254) 1668 : cluster [DBG] pgmap v1361: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:21.024 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:44:22.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:21 smithi132 bash[18607]: cluster 2024-04-25T20:44:21.918128+0000 mgr.x (mgr.14254) 1669 : cluster [DBG] pgmap v1362: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:22.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:21 smithi132 bash[18607]: cluster 2024-04-25T20:44:21.918128+0000 mgr.x (mgr.14254) 1669 : cluster [DBG] pgmap v1362: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:24.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:23 smithi132 bash[18607]: cluster 2024-04-25T20:44:23.920173+0000 mgr.x (mgr.14254) 1670 : cluster [DBG] pgmap v1363: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:24.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:23 smithi132 bash[18607]: cluster 2024-04-25T20:44:23.920173+0000 mgr.x (mgr.14254) 1670 : cluster [DBG] pgmap v1363: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:25.783 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:44:26.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:25 smithi132 bash[18607]: cluster 2024-04-25T20:44:25.921797+0000 mgr.x (mgr.14254) 1671 : cluster [DBG] pgmap v1364: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:26.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:25 smithi132 bash[18607]: cluster 2024-04-25T20:44:25.921797+0000 mgr.x (mgr.14254) 1671 : cluster [DBG] pgmap v1364: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:28.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:27 smithi132 bash[18607]: cluster 2024-04-25T20:44:27.923675+0000 mgr.x (mgr.14254) 1672 : cluster [DBG] pgmap v1365: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:28.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:27 smithi132 bash[18607]: cluster 2024-04-25T20:44:27.923675+0000 mgr.x (mgr.14254) 1672 : cluster [DBG] pgmap v1365: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:28.692 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:44:28.692 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:44:29.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:28 smithi132 bash[18607]: audit 2024-04-25T20:44:28.692540+0000 mgr.x (mgr.14254) 1673 : audit [DBG] from='client.15040 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:44:29.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:28 smithi132 bash[18607]: audit 2024-04-25T20:44:28.692540+0000 mgr.x (mgr.14254) 1673 : audit [DBG] from='client.15040 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:44:29.395 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:44:30.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:30 smithi132 bash[18607]: cluster 2024-04-25T20:44:29.925197+0000 mgr.x (mgr.14254) 1674 : cluster [DBG] pgmap v1366: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:30.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:30 smithi132 bash[18607]: cluster 2024-04-25T20:44:29.925197+0000 mgr.x (mgr.14254) 1674 : cluster [DBG] pgmap v1366: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:30.397 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:44:32.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:31 smithi132 bash[18607]: cluster 2024-04-25T20:44:31.926640+0000 mgr.x (mgr.14254) 1675 : cluster [DBG] pgmap v1367: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:32.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:31 smithi132 bash[18607]: cluster 2024-04-25T20:44:31.926640+0000 mgr.x (mgr.14254) 1675 : cluster [DBG] pgmap v1367: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:34.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:33 smithi132 bash[18607]: cluster 2024-04-25T20:44:33.928560+0000 mgr.x (mgr.14254) 1676 : cluster [DBG] pgmap v1368: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:34.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:33 smithi132 bash[18607]: cluster 2024-04-25T20:44:33.928560+0000 mgr.x (mgr.14254) 1676 : cluster [DBG] pgmap v1368: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:35.154 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:44:36.258 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:35 smithi132 bash[18607]: cluster 2024-04-25T20:44:35.929963+0000 mgr.x (mgr.14254) 1677 : cluster [DBG] pgmap v1369: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:36.259 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:35 smithi132 bash[18607]: cluster 2024-04-25T20:44:35.929963+0000 mgr.x (mgr.14254) 1677 : cluster [DBG] pgmap v1369: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:37.909 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:44:37.909 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:44:38.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:37 smithi132 bash[18607]: audit 2024-04-25T20:44:37.907877+0000 mgr.x (mgr.14254) 1678 : audit [DBG] from='client.15042 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:44:38.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:37 smithi132 bash[18607]: audit 2024-04-25T20:44:37.907877+0000 mgr.x (mgr.14254) 1678 : audit [DBG] from='client.15042 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:44:38.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:37 smithi132 bash[18607]: cluster 2024-04-25T20:44:37.931247+0000 mgr.x (mgr.14254) 1679 : cluster [DBG] pgmap v1370: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:38.273 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:37 smithi132 bash[18607]: cluster 2024-04-25T20:44:37.931247+0000 mgr.x (mgr.14254) 1679 : cluster [DBG] pgmap v1370: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:38.632 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:44:39.633 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:44:40.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:39 smithi132 bash[18607]: cluster 2024-04-25T20:44:39.932608+0000 mgr.x (mgr.14254) 1680 : cluster [DBG] pgmap v1371: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:40.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:39 smithi132 bash[18607]: cluster 2024-04-25T20:44:39.932608+0000 mgr.x (mgr.14254) 1680 : cluster [DBG] pgmap v1371: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:42.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:41 smithi132 bash[18607]: cluster 2024-04-25T20:44:41.934038+0000 mgr.x (mgr.14254) 1681 : cluster [DBG] pgmap v1372: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:42.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:41 smithi132 bash[18607]: cluster 2024-04-25T20:44:41.934038+0000 mgr.x (mgr.14254) 1681 : cluster [DBG] pgmap v1372: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:44.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:43 smithi132 bash[18607]: cluster 2024-04-25T20:44:43.935941+0000 mgr.x (mgr.14254) 1682 : cluster [DBG] pgmap v1373: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:44.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:43 smithi132 bash[18607]: cluster 2024-04-25T20:44:43.935941+0000 mgr.x (mgr.14254) 1682 : cluster [DBG] pgmap v1373: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:44.390 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:44:46.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:45 smithi132 bash[18607]: cluster 2024-04-25T20:44:45.937601+0000 mgr.x (mgr.14254) 1683 : cluster [DBG] pgmap v1374: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:46.274 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:45 smithi132 bash[18607]: cluster 2024-04-25T20:44:45.937601+0000 mgr.x (mgr.14254) 1683 : cluster [DBG] pgmap v1374: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:47.255 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:44:47.255 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:44:47.941 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:47 smithi132 bash[18607]: audit 2024-04-25T20:44:47.546229+0000 mon.a (mon.0) 532 : audit [DBG] from='client.? 172.17.0.2:0/3990797609' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:44:47.941 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:47 smithi132 bash[18607]: audit 2024-04-25T20:44:47.546229+0000 mon.a (mon.0) 532 : audit [DBG] from='client.? 172.17.0.2:0/3990797609' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:44:47.942 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:44:48.943 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:44:48.955 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:48 smithi132 bash[18607]: audit 2024-04-25T20:44:47.253884+0000 mgr.x (mgr.14254) 1684 : audit [DBG] from='client.15044 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:44:48.955 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:48 smithi132 bash[18607]: audit 2024-04-25T20:44:47.253884+0000 mgr.x (mgr.14254) 1684 : audit [DBG] from='client.15044 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:44:48.955 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:48 smithi132 bash[18607]: cluster 2024-04-25T20:44:47.939662+0000 mgr.x (mgr.14254) 1685 : cluster [DBG] pgmap v1375: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:48.955 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:48 smithi132 bash[18607]: cluster 2024-04-25T20:44:47.939662+0000 mgr.x (mgr.14254) 1685 : cluster [DBG] pgmap v1375: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:50.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:49 smithi132 bash[18607]: cluster 2024-04-25T20:44:49.941129+0000 mgr.x (mgr.14254) 1686 : cluster [DBG] pgmap v1376: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:50.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:49 smithi132 bash[18607]: cluster 2024-04-25T20:44:49.941129+0000 mgr.x (mgr.14254) 1686 : cluster [DBG] pgmap v1376: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:53.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:52 smithi132 bash[18607]: cluster 2024-04-25T20:44:51.942557+0000 mgr.x (mgr.14254) 1687 : cluster [DBG] pgmap v1377: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:53.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:52 smithi132 bash[18607]: cluster 2024-04-25T20:44:51.942557+0000 mgr.x (mgr.14254) 1687 : cluster [DBG] pgmap v1377: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:53.703 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:44:55.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:54 smithi132 bash[18607]: cluster 2024-04-25T20:44:53.944414+0000 mgr.x (mgr.14254) 1688 : cluster [DBG] pgmap v1378: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:55.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:54 smithi132 bash[18607]: cluster 2024-04-25T20:44:53.944414+0000 mgr.x (mgr.14254) 1688 : cluster [DBG] pgmap v1378: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:56.660 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:44:56.661 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:44:57.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:57 smithi132 bash[18607]: cluster 2024-04-25T20:44:55.945850+0000 mgr.x (mgr.14254) 1689 : cluster [DBG] pgmap v1379: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:57.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:57 smithi132 bash[18607]: cluster 2024-04-25T20:44:55.945850+0000 mgr.x (mgr.14254) 1689 : cluster [DBG] pgmap v1379: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:57.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:57 smithi132 bash[18607]: audit 2024-04-25T20:44:56.659421+0000 mgr.x (mgr.14254) 1690 : audit [DBG] from='client.15048 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:44:57.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:57 smithi132 bash[18607]: audit 2024-04-25T20:44:56.659421+0000 mgr.x (mgr.14254) 1690 : audit [DBG] from='client.15048 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:44:57.414 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:44:58.415 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:44:59.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:59 smithi132 bash[18607]: cluster 2024-04-25T20:44:57.947661+0000 mgr.x (mgr.14254) 1691 : cluster [DBG] pgmap v1380: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:44:59.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:44:59 smithi132 bash[18607]: cluster 2024-04-25T20:44:57.947661+0000 mgr.x (mgr.14254) 1691 : cluster [DBG] pgmap v1380: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:01.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:01 smithi132 bash[18607]: cluster 2024-04-25T20:44:59.949114+0000 mgr.x (mgr.14254) 1692 : cluster [DBG] pgmap v1381: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:01.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:01 smithi132 bash[18607]: cluster 2024-04-25T20:44:59.949114+0000 mgr.x (mgr.14254) 1692 : cluster [DBG] pgmap v1381: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:03.177 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:45:03.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:03 smithi132 bash[18607]: cluster 2024-04-25T20:45:01.950493+0000 mgr.x (mgr.14254) 1693 : cluster [DBG] pgmap v1382: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:03.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:03 smithi132 bash[18607]: cluster 2024-04-25T20:45:01.950493+0000 mgr.x (mgr.14254) 1693 : cluster [DBG] pgmap v1382: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:05.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:05 smithi132 bash[18607]: cluster 2024-04-25T20:45:03.952342+0000 mgr.x (mgr.14254) 1694 : cluster [DBG] pgmap v1383: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:05.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:05 smithi132 bash[18607]: cluster 2024-04-25T20:45:03.952342+0000 mgr.x (mgr.14254) 1694 : cluster [DBG] pgmap v1383: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:06.247 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:45:06.247 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:45:07.020 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:45:07.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:07 smithi132 bash[18607]: cluster 2024-04-25T20:45:05.953289+0000 mgr.x (mgr.14254) 1695 : cluster [DBG] pgmap v1384: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:07.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:07 smithi132 bash[18607]: cluster 2024-04-25T20:45:05.953289+0000 mgr.x (mgr.14254) 1695 : cluster [DBG] pgmap v1384: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:07.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:07 smithi132 bash[18607]: audit 2024-04-25T20:45:06.249559+0000 mgr.x (mgr.14254) 1696 : audit [DBG] from='client.15050 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:45:07.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:07 smithi132 bash[18607]: audit 2024-04-25T20:45:06.249559+0000 mgr.x (mgr.14254) 1696 : audit [DBG] from='client.15050 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:45:08.021 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:45:09.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:09 smithi132 bash[18607]: cluster 2024-04-25T20:45:07.955138+0000 mgr.x (mgr.14254) 1697 : cluster [DBG] pgmap v1385: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:09.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:09 smithi132 bash[18607]: cluster 2024-04-25T20:45:07.955138+0000 mgr.x (mgr.14254) 1697 : cluster [DBG] pgmap v1385: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:11.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:11 smithi132 bash[18607]: cluster 2024-04-25T20:45:09.956515+0000 mgr.x (mgr.14254) 1698 : cluster [DBG] pgmap v1386: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:11.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:11 smithi132 bash[18607]: cluster 2024-04-25T20:45:09.956515+0000 mgr.x (mgr.14254) 1698 : cluster [DBG] pgmap v1386: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:12.768 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:45:13.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:13 smithi132 bash[18607]: cluster 2024-04-25T20:45:11.957907+0000 mgr.x (mgr.14254) 1699 : cluster [DBG] pgmap v1387: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:13.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:13 smithi132 bash[18607]: cluster 2024-04-25T20:45:11.957907+0000 mgr.x (mgr.14254) 1699 : cluster [DBG] pgmap v1387: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:15.264 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:15 smithi132 bash[18607]: cluster 2024-04-25T20:45:13.959119+0000 mgr.x (mgr.14254) 1700 : cluster [DBG] pgmap v1388: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:15.264 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:15 smithi132 bash[18607]: cluster 2024-04-25T20:45:13.959119+0000 mgr.x (mgr.14254) 1700 : cluster [DBG] pgmap v1388: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:15.610 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:45:15.611 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:45:16.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:16 smithi132 bash[18607]: audit 2024-04-25T20:45:15.609606+0000 mgr.x (mgr.14254) 1701 : audit [DBG] from='client.15052 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:45:16.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:16 smithi132 bash[18607]: audit 2024-04-25T20:45:15.609606+0000 mgr.x (mgr.14254) 1701 : audit [DBG] from='client.15052 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:45:16.327 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:45:17.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:17 smithi132 bash[18607]: cluster 2024-04-25T20:45:15.960445+0000 mgr.x (mgr.14254) 1702 : cluster [DBG] pgmap v1389: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:17.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:17 smithi132 bash[18607]: cluster 2024-04-25T20:45:15.960445+0000 mgr.x (mgr.14254) 1702 : cluster [DBG] pgmap v1389: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:17.328 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:45:19.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:19 smithi132 bash[18607]: cluster 2024-04-25T20:45:17.962243+0000 mgr.x (mgr.14254) 1703 : cluster [DBG] pgmap v1390: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:19.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:19 smithi132 bash[18607]: cluster 2024-04-25T20:45:17.962243+0000 mgr.x (mgr.14254) 1703 : cluster [DBG] pgmap v1390: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:21.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:21 smithi132 bash[18607]: cluster 2024-04-25T20:45:19.963632+0000 mgr.x (mgr.14254) 1704 : cluster [DBG] pgmap v1391: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:21.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:21 smithi132 bash[18607]: cluster 2024-04-25T20:45:19.963632+0000 mgr.x (mgr.14254) 1704 : cluster [DBG] pgmap v1391: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:22.087 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:45:23.243 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:23 smithi132 bash[18607]: cluster 2024-04-25T20:45:21.965088+0000 mgr.x (mgr.14254) 1705 : cluster [DBG] pgmap v1392: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:23.243 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:23 smithi132 bash[18607]: cluster 2024-04-25T20:45:21.965088+0000 mgr.x (mgr.14254) 1705 : cluster [DBG] pgmap v1392: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:25.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:25 smithi132 bash[18607]: cluster 2024-04-25T20:45:23.966961+0000 mgr.x (mgr.14254) 1706 : cluster [DBG] pgmap v1393: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:25.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:25 smithi132 bash[18607]: cluster 2024-04-25T20:45:23.966961+0000 mgr.x (mgr.14254) 1706 : cluster [DBG] pgmap v1393: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:25.274 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:45:25.275 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:45:26.030 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:45:26.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:26 smithi132 bash[18607]: audit 2024-04-25T20:45:25.273656+0000 mgr.x (mgr.14254) 1707 : audit [DBG] from='client.15054 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:45:26.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:26 smithi132 bash[18607]: audit 2024-04-25T20:45:25.273656+0000 mgr.x (mgr.14254) 1707 : audit [DBG] from='client.15054 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:45:27.031 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:45:27.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:27 smithi132 bash[18607]: cluster 2024-04-25T20:45:25.968338+0000 mgr.x (mgr.14254) 1708 : cluster [DBG] pgmap v1394: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:27.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:27 smithi132 bash[18607]: cluster 2024-04-25T20:45:25.968338+0000 mgr.x (mgr.14254) 1708 : cluster [DBG] pgmap v1394: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:29.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:29 smithi132 bash[18607]: cluster 2024-04-25T20:45:27.970129+0000 mgr.x (mgr.14254) 1709 : cluster [DBG] pgmap v1395: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:29.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:29 smithi132 bash[18607]: cluster 2024-04-25T20:45:27.970129+0000 mgr.x (mgr.14254) 1709 : cluster [DBG] pgmap v1395: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:31.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:31 smithi132 bash[18607]: cluster 2024-04-25T20:45:29.971547+0000 mgr.x (mgr.14254) 1710 : cluster [DBG] pgmap v1396: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:31.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:31 smithi132 bash[18607]: cluster 2024-04-25T20:45:29.971547+0000 mgr.x (mgr.14254) 1710 : cluster [DBG] pgmap v1396: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:31.785 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:45:33.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:33 smithi132 bash[18607]: cluster 2024-04-25T20:45:31.973015+0000 mgr.x (mgr.14254) 1711 : cluster [DBG] pgmap v1397: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:33.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:33 smithi132 bash[18607]: cluster 2024-04-25T20:45:31.973015+0000 mgr.x (mgr.14254) 1711 : cluster [DBG] pgmap v1397: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:34.685 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:45:34.685 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:45:35.398 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:35 smithi132 bash[18607]: cluster 2024-04-25T20:45:33.974800+0000 mgr.x (mgr.14254) 1712 : cluster [DBG] pgmap v1398: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:35.398 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:35 smithi132 bash[18607]: cluster 2024-04-25T20:45:33.974800+0000 mgr.x (mgr.14254) 1712 : cluster [DBG] pgmap v1398: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:35.398 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:35 smithi132 bash[18607]: audit 2024-04-25T20:45:34.684052+0000 mgr.x (mgr.14254) 1713 : audit [DBG] from='client.15056 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:45:35.399 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:35 smithi132 bash[18607]: audit 2024-04-25T20:45:34.684052+0000 mgr.x (mgr.14254) 1713 : audit [DBG] from='client.15056 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:45:35.399 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:45:36.401 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:45:37.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:37 smithi132 bash[18607]: cluster 2024-04-25T20:45:35.976257+0000 mgr.x (mgr.14254) 1714 : cluster [DBG] pgmap v1399: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:37.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:37 smithi132 bash[18607]: cluster 2024-04-25T20:45:35.976257+0000 mgr.x (mgr.14254) 1714 : cluster [DBG] pgmap v1399: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:39.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:39 smithi132 bash[18607]: cluster 2024-04-25T20:45:37.978228+0000 mgr.x (mgr.14254) 1715 : cluster [DBG] pgmap v1400: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:39.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:39 smithi132 bash[18607]: cluster 2024-04-25T20:45:37.978228+0000 mgr.x (mgr.14254) 1715 : cluster [DBG] pgmap v1400: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:41.164 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:45:41.523 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:41 smithi132 bash[18607]: cluster 2024-04-25T20:45:39.979739+0000 mgr.x (mgr.14254) 1716 : cluster [DBG] pgmap v1401: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:41.524 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:41 smithi132 bash[18607]: cluster 2024-04-25T20:45:39.979739+0000 mgr.x (mgr.14254) 1716 : cluster [DBG] pgmap v1401: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:43.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:43 smithi132 bash[18607]: cluster 2024-04-25T20:45:41.981177+0000 mgr.x (mgr.14254) 1717 : cluster [DBG] pgmap v1402: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:43.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:43 smithi132 bash[18607]: cluster 2024-04-25T20:45:41.981177+0000 mgr.x (mgr.14254) 1717 : cluster [DBG] pgmap v1402: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:43.973 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:45:43.973 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:45:44.723 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:45:45.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:45 smithi132 bash[18607]: audit 2024-04-25T20:45:43.971458+0000 mgr.x (mgr.14254) 1718 : audit [DBG] from='client.15058 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:45:45.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:45 smithi132 bash[18607]: audit 2024-04-25T20:45:43.971458+0000 mgr.x (mgr.14254) 1718 : audit [DBG] from='client.15058 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:45:45.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:45 smithi132 bash[18607]: cluster 2024-04-25T20:45:43.982203+0000 mgr.x (mgr.14254) 1719 : cluster [DBG] pgmap v1403: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:45.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:45 smithi132 bash[18607]: cluster 2024-04-25T20:45:43.982203+0000 mgr.x (mgr.14254) 1719 : cluster [DBG] pgmap v1403: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:45.724 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:45:47.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:47 smithi132 bash[18607]: cluster 2024-04-25T20:45:45.983459+0000 mgr.x (mgr.14254) 1720 : cluster [DBG] pgmap v1404: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:47.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:47 smithi132 bash[18607]: cluster 2024-04-25T20:45:45.983459+0000 mgr.x (mgr.14254) 1720 : cluster [DBG] pgmap v1404: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:49.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:49 smithi132 bash[18607]: cluster 2024-04-25T20:45:47.985412+0000 mgr.x (mgr.14254) 1721 : cluster [DBG] pgmap v1405: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:49.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:49 smithi132 bash[18607]: cluster 2024-04-25T20:45:47.985412+0000 mgr.x (mgr.14254) 1721 : cluster [DBG] pgmap v1405: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:50.484 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:45:51.325 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:51 smithi132 bash[18607]: cluster 2024-04-25T20:45:49.986555+0000 mgr.x (mgr.14254) 1722 : cluster [DBG] pgmap v1406: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:51.325 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:51 smithi132 bash[18607]: cluster 2024-04-25T20:45:49.986555+0000 mgr.x (mgr.14254) 1722 : cluster [DBG] pgmap v1406: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:53.210 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:45:53.210 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:45:53.422 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:53 smithi132 bash[18607]: cluster 2024-04-25T20:45:51.987959+0000 mgr.x (mgr.14254) 1723 : cluster [DBG] pgmap v1407: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:53.422 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:53 smithi132 bash[18607]: cluster 2024-04-25T20:45:51.987959+0000 mgr.x (mgr.14254) 1723 : cluster [DBG] pgmap v1407: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:53.963 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:45:54.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:54 smithi132 bash[18607]: audit 2024-04-25T20:45:53.209107+0000 mgr.x (mgr.14254) 1724 : audit [DBG] from='client.15060 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:45:54.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:54 smithi132 bash[18607]: audit 2024-04-25T20:45:53.209107+0000 mgr.x (mgr.14254) 1724 : audit [DBG] from='client.15060 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:45:54.964 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:45:55.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:55 smithi132 bash[18607]: cluster 2024-04-25T20:45:53.989890+0000 mgr.x (mgr.14254) 1725 : cluster [DBG] pgmap v1408: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:55.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:55 smithi132 bash[18607]: cluster 2024-04-25T20:45:53.989890+0000 mgr.x (mgr.14254) 1725 : cluster [DBG] pgmap v1408: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:57.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:57 smithi132 bash[18607]: cluster 2024-04-25T20:45:55.991246+0000 mgr.x (mgr.14254) 1726 : cluster [DBG] pgmap v1409: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:57.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:57 smithi132 bash[18607]: cluster 2024-04-25T20:45:55.991246+0000 mgr.x (mgr.14254) 1726 : cluster [DBG] pgmap v1409: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:58.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:58 smithi132 bash[18607]: audit 2024-04-25T20:45:57.930869+0000 mon.a (mon.0) 533 : audit [DBG] from='client.? 172.17.0.2:0/3927339747' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:45:58.271 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:58 smithi132 bash[18607]: audit 2024-04-25T20:45:57.930869+0000 mon.a (mon.0) 533 : audit [DBG] from='client.? 172.17.0.2:0/3927339747' entity='client.smb.config.modtest1.smithi132.jrsyig' cmd=[{"prefix": "config-key get", "key": "smb/config/modtest1/join.0.json"}]: dispatch 2024-04-25T20:45:59.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:59 smithi132 bash[18607]: cluster 2024-04-25T20:45:57.992777+0000 mgr.x (mgr.14254) 1727 : cluster [DBG] pgmap v1410: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:59.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:45:59 smithi132 bash[18607]: cluster 2024-04-25T20:45:57.992777+0000 mgr.x (mgr.14254) 1727 : cluster [DBG] pgmap v1410: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:45:59.728 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:46:01.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:46:01 smithi132 bash[18607]: cluster 2024-04-25T20:45:59.994306+0000 mgr.x (mgr.14254) 1728 : cluster [DBG] pgmap v1411: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:46:01.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:46:01 smithi132 bash[18607]: cluster 2024-04-25T20:45:59.994306+0000 mgr.x (mgr.14254) 1728 : cluster [DBG] pgmap v1411: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:46:02.574 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:46:02.574 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:46:03.307 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:46:03.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:46:03 smithi132 bash[18607]: cluster 2024-04-25T20:46:01.995887+0000 mgr.x (mgr.14254) 1729 : cluster [DBG] pgmap v1412: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:46:03.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:46:03 smithi132 bash[18607]: cluster 2024-04-25T20:46:01.995887+0000 mgr.x (mgr.14254) 1729 : cluster [DBG] pgmap v1412: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:46:03.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:46:03 smithi132 bash[18607]: audit 2024-04-25T20:46:02.572719+0000 mgr.x (mgr.14254) 1730 : audit [DBG] from='client.15064 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:46:03.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:46:03 smithi132 bash[18607]: audit 2024-04-25T20:46:02.572719+0000 mgr.x (mgr.14254) 1730 : audit [DBG] from='client.15064 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:46:04.307 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:46:05.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:46:05 smithi132 bash[18607]: cluster 2024-04-25T20:46:03.997777+0000 mgr.x (mgr.14254) 1731 : cluster [DBG] pgmap v1413: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:46:05.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:46:05 smithi132 bash[18607]: cluster 2024-04-25T20:46:03.997777+0000 mgr.x (mgr.14254) 1731 : cluster [DBG] pgmap v1413: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:46:07.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:46:07 smithi132 bash[18607]: cluster 2024-04-25T20:46:05.999110+0000 mgr.x (mgr.14254) 1732 : cluster [DBG] pgmap v1414: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:46:07.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:46:07 smithi132 bash[18607]: cluster 2024-04-25T20:46:05.999110+0000 mgr.x (mgr.14254) 1732 : cluster [DBG] pgmap v1414: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:46:09.064 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:46:09.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:46:09 smithi132 bash[18607]: cluster 2024-04-25T20:46:07.999981+0000 mgr.x (mgr.14254) 1733 : cluster [DBG] pgmap v1415: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:46:09.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:46:09 smithi132 bash[18607]: cluster 2024-04-25T20:46:07.999981+0000 mgr.x (mgr.14254) 1733 : cluster [DBG] pgmap v1415: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:46:11.486 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:46:11 smithi132 bash[18607]: cluster 2024-04-25T20:46:10.000720+0000 mgr.x (mgr.14254) 1734 : cluster [DBG] pgmap v1416: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:46:11.486 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:46:11 smithi132 bash[18607]: cluster 2024-04-25T20:46:10.000720+0000 mgr.x (mgr.14254) 1734 : cluster [DBG] pgmap v1416: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:46:11.872 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:46:11.872 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:46:12.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:46:12 smithi132 bash[18607]: audit 2024-04-25T20:46:11.874232+0000 mgr.x (mgr.14254) 1735 : audit [DBG] from='client.15066 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:46:12.270 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:46:12 smithi132 bash[18607]: audit 2024-04-25T20:46:11.874232+0000 mgr.x (mgr.14254) 1735 : audit [DBG] from='client.15066 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:46:12.737 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:46:13.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:46:13 smithi132 bash[18607]: cluster 2024-04-25T20:46:12.002010+0000 mgr.x (mgr.14254) 1736 : cluster [DBG] pgmap v1417: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:46:13.522 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:46:13 smithi132 bash[18607]: cluster 2024-04-25T20:46:12.002010+0000 mgr.x (mgr.14254) 1736 : cluster [DBG] pgmap v1417: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:46:13.737 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b48599b46c4ce29785962a8af5ddf30f6f38e57f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3114c848-033d-11ef-bc93-c7b262605968 -- ceph orch ls -f json 2024-04-25T20:46:15.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:46:15 smithi132 bash[18607]: cluster 2024-04-25T20:46:14.003814+0000 mgr.x (mgr.14254) 1737 : cluster [DBG] pgmap v1418: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:46:15.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:46:15 smithi132 bash[18607]: cluster 2024-04-25T20:46:14.003814+0000 mgr.x (mgr.14254) 1737 : cluster [DBG] pgmap v1418: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:46:17.520 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:46:17 smithi132 bash[18607]: cluster 2024-04-25T20:46:16.005268+0000 mgr.x (mgr.14254) 1738 : cluster [DBG] pgmap v1419: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:46:17.521 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:46:17 smithi132 bash[18607]: cluster 2024-04-25T20:46:16.005268+0000 mgr.x (mgr.14254) 1738 : cluster [DBG] pgmap v1419: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:46:18.500 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T20:46:19.424 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:46:19 smithi132 bash[18607]: cluster 2024-04-25T20:46:18.006995+0000 mgr.x (mgr.14254) 1739 : cluster [DBG] pgmap v1420: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:46:19.424 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:46:19 smithi132 bash[18607]: cluster 2024-04-25T20:46:18.006995+0000 mgr.x (mgr.14254) 1739 : cluster [DBG] pgmap v1420: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:46:21.436 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-04-25T20:46:21.436 INFO:teuthology.orchestra.run.smithi132.stdout:[{"placement": {"count": 2}, "service_id": "cephfs", "service_name": "mds.cephfs", "service_type": "mds", "status": {"created": "2024-04-25T19:58:12.941580Z", "last_refresh": "2024-04-25T20:31:54.932377Z", "running": 2, "size": 2}}, {"placement": {"count": 1, "hosts": ["smithi132=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-25T19:54:52.932235Z", "last_refresh": "2024-04-25T20:31:54.931777Z", "running": 1, "size": 1}}, {"placement": {"count": 1, "hosts": ["smithi132:172.21.15.132=a"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-25T19:54:28.714229Z", "last_refresh": "2024-04-25T20:31:54.931646Z", "running": 1, "size": 1}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "2622ebd426adde4a016357d27d58ff34de90d8c4cafb9e55af4079c6c116943b", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:26ccac2819be74cc0ff713c4e4e866c3943252a2928101911aaf7cfe721c581a", "last_refresh": "2024-04-25T20:31:54.931890Z", "running": 2, "size": 2}, "unmanaged": true}, {"events": ["2024-04-25T20:14:45.988887Z service:smb.modtest1 [INFO] \"service was created\"", "2024-04-25T20:14:45.981136Z service:smb.modtest1 [ERROR] \"Failed while placing smb.modtest1.smithi132.jrsyig on smithi132: cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb-modtest1-smithi132-jrsyig\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-smb.modtest1.smithi132.jrsyig\nDeploy daemon smb.modtest1.smithi132.jrsyig ...\nVerifying port 0.0.0.0:445 ...\nsystemctl: timeout after 895 seconds\nNon-zero exit code 124 from systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\nTraceback (most recent call last):\n File \"/usr/lib/python3.10/runpy.py\", line 196, in _run_module_as_main\n return _run_code(code, main_globals, None,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5618, in \n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 5606, in main\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3096, in command_deploy_from\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3131, in _common_deploy\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 3151, in _deploy_daemon_container\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1087, in deploy_daemon\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/__main__.py\", line 1223, in deploy_daemon_units\n File \"/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/cephadm.e84f0e25c6519f46968a4b9cf4942f78ad786b9ee9a1980ce9d880a27a8aa7bc/cephadmlib/call_wrappers.py\", line 310, in call_throws\nRuntimeError: Failed command: systemctl start ceph-3114c848-033d-11ef-bc93-c7b262605968@smb.modtest1.smithi132.jrsyig\""], "placement": {"count": 1}, "service_id": "modtest1", "service_name": "smb.modtest1", "service_type": "smb", "spec": {"cluster_id": "modtest1", "config_uri": "rados://.smb/modtest1/config.smb", "custom_dns": ["192.168.76.204"], "features": ["domain"], "include_ceph_users": ["client.smb.fs.cluster.modtest1"], "join_sources": ["rados:mon-config-key:smb/config/modtest1/join.0.json"]}, "status": {"created": "2024-04-25T19:59:49.408077Z", "last_refresh": "2024-04-25T20:31:54.932906Z", "running": 0, "size": 1}}] 2024-04-25T20:46:21.449 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:46:21 smithi132 bash[18607]: cluster 2024-04-25T20:46:20.008377+0000 mgr.x (mgr.14254) 1740 : cluster [DBG] pgmap v1421: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:46:21.449 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:46:21 smithi132 bash[18607]: cluster 2024-04-25T20:46:20.008377+0000 mgr.x (mgr.14254) 1740 : cluster [DBG] pgmap v1421: 96 pgs: 18 active+undersized+degraded, 78 active+undersized; 68 KiB data, 57 MiB used, 179 GiB / 179 GiB avail; 27/81 objects degraded (33.333%) 2024-04-25T20:46:22.247 INFO:tasks.cephadm:smb.modtest1 has 0/1 2024-04-25T20:46:22.247 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_b48599b46c4ce29785962a8af5ddf30f6f38e57f/qa/tasks/cephadm.py", line 1592, in wait_for_service while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (301) after waiting for 300 seconds 2024-04-25T20:46:22.455 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=bfb1dfe28a1741d19b1a00c3492edbc9 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_b48599b46c4ce29785962a8af5ddf30f6f38e57f/qa/tasks/cephadm.py", line 1592, in wait_for_service while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (301) after waiting for 300 seconds 2024-04-25T20:46:22.457 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-25T20:46:22.467 INFO:tasks.cephadm:Teardown begin 2024-04-25T20:46:22.468 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T20:46:22.483 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:46:22 smithi132 bash[18607]: audit 2024-04-25T20:46:21.437727+0000 mgr.x (mgr.14254) 1741 : audit [DBG] from='client.15068 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:46:22.483 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:46:22 smithi132 bash[18607]: audit 2024-04-25T20:46:21.437727+0000 mgr.x (mgr.14254) 1741 : audit [DBG] from='client.15068 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T20:46:22.485 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T20:46:22.500 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-25T20:46:22.500 DEBUG:teuthology.orchestra.run.smithi132:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-25T20:46:22.532 DEBUG:teuthology.orchestra.run.smithi150:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-25T20:46:22.549 INFO:tasks.cephadm:Stopping all daemons... 2024-04-25T20:46:22.549 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-25T20:46:22.549 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-3114c848-033d-11ef-bc93-c7b262605968@mon.a 2024-04-25T20:46:22.841 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:46:22 smithi132 systemd[1]: Stopping Ceph mon.a for 3114c848-033d-11ef-bc93-c7b262605968... 2024-04-25T20:46:22.841 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:46:22 smithi132 bash[18607]: debug 2024-04-25T20:46:22.714+0000 7fdd03e8a700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-04-25T20:46:22.841 INFO:journalctl@ceph.mon.a.smithi132.stdout:Apr 25 20:46:22 smithi132 bash[18607]: debug 2024-04-25T20:46:22.714+0000 7fdd03e8a700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-25T20:46:23.691 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3114c848-033d-11ef-bc93-c7b262605968@mon.a.service' 2024-04-25T20:46:23.721 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T20:46:23.721 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-25T20:46:23.721 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-25T20:46:23.721 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-3114c848-033d-11ef-bc93-c7b262605968@mgr.x 2024-04-25T20:46:23.940 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:46:23 smithi132 systemd[1]: Stopping Ceph mgr.x for 3114c848-033d-11ef-bc93-c7b262605968... 2024-04-25T20:46:23.941 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:46:23 smithi132 bash[18860]: debug 2024-04-25T20:46:23.898+0000 7f9182c30700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-04-25T20:46:23.941 INFO:journalctl@ceph.mgr.x.smithi132.stdout:Apr 25 20:46:23 smithi132 bash[18860]: debug 2024-04-25T20:46:23.898+0000 7f9182c30700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-25T20:46:24.761 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3114c848-033d-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-25T20:46:24.784 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T20:46:24.784 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-25T20:46:24.784 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-25T20:46:24.784 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-3114c848-033d-11ef-bc93-c7b262605968@osd.0 2024-04-25T20:46:25.021 INFO:journalctl@ceph.osd.0.smithi132.stdout:Apr 25 20:46:24 smithi132 systemd[1]: Stopping Ceph osd.0 for 3114c848-033d-11ef-bc93-c7b262605968... 2024-04-25T20:46:25.021 INFO:journalctl@ceph.osd.0.smithi132.stdout:Apr 25 20:46:24 smithi132 bash[28612]: debug 2024-04-25T20:46:24.958+0000 7f06dc3d3700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-04-25T20:46:25.021 INFO:journalctl@ceph.osd.0.smithi132.stdout:Apr 25 20:46:24 smithi132 bash[28612]: debug 2024-04-25T20:46:24.958+0000 7f06dc3d3700 -1 osd.0 24 *** Got signal Terminated *** 2024-04-25T20:46:25.021 INFO:journalctl@ceph.osd.0.smithi132.stdout:Apr 25 20:46:24 smithi132 bash[28612]: debug 2024-04-25T20:46:24.958+0000 7f06dc3d3700 -1 osd.0 24 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T20:46:31.520 INFO:journalctl@ceph.osd.0.smithi132.stdout:Apr 25 20:46:31 smithi132 bash[112041]: ceph-3114c848-033d-11ef-bc93-c7b262605968-osd-0 2024-04-25T20:46:31.521 INFO:journalctl@ceph.osd.0.smithi132.stdout:Apr 25 20:46:31 smithi132 bash[112101]: ceph-3114c848-033d-11ef-bc93-c7b262605968-osd-0 2024-04-25T20:46:33.228 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3114c848-033d-11ef-bc93-c7b262605968@osd.0.service' 2024-04-25T20:46:33.254 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T20:46:33.255 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-25T20:46:33.255 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-25T20:46:33.255 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-3114c848-033d-11ef-bc93-c7b262605968@osd.1 2024-04-25T20:46:33.521 INFO:journalctl@ceph.osd.1.smithi132.stdout:Apr 25 20:46:33 smithi132 systemd[1]: Stopping Ceph osd.1 for 3114c848-033d-11ef-bc93-c7b262605968... 2024-04-25T20:46:33.521 INFO:journalctl@ceph.osd.1.smithi132.stdout:Apr 25 20:46:33 smithi132 bash[33896]: debug 2024-04-25T20:46:33.438+0000 7fb300ed0700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-04-25T20:46:33.521 INFO:journalctl@ceph.osd.1.smithi132.stdout:Apr 25 20:46:33 smithi132 bash[33896]: debug 2024-04-25T20:46:33.438+0000 7fb300ed0700 -1 osd.1 24 *** Got signal Terminated *** 2024-04-25T20:46:33.521 INFO:journalctl@ceph.osd.1.smithi132.stdout:Apr 25 20:46:33 smithi132 bash[33896]: debug 2024-04-25T20:46:33.438+0000 7fb300ed0700 -1 osd.1 24 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T20:46:40.021 INFO:journalctl@ceph.osd.1.smithi132.stdout:Apr 25 20:46:39 smithi132 bash[112235]: ceph-3114c848-033d-11ef-bc93-c7b262605968-osd-1 2024-04-25T20:46:40.021 INFO:journalctl@ceph.osd.1.smithi132.stdout:Apr 25 20:46:39 smithi132 bash[112296]: Error response from daemon: No such container: ceph-3114c848-033d-11ef-bc93-c7b262605968-osd-1 2024-04-25T20:46:41.628 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3114c848-033d-11ef-bc93-c7b262605968@osd.1.service' 2024-04-25T20:46:41.661 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T20:46:41.662 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-25T20:46:41.662 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3114c848-033d-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-25T20:47:00.926 INFO:teuthology.orchestra.run.smithi132.stdout:Deleting cluster with fsid: 3114c848-033d-11ef-bc93-c7b262605968 2024-04-25T20:47:32.633 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3114c848-033d-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-25T20:47:32.801 INFO:teuthology.orchestra.run.smithi150.stdout:Deleting cluster with fsid: 3114c848-033d-11ef-bc93-c7b262605968 2024-04-25T20:47:36.429 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T20:47:36.443 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T20:47:36.457 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-25T20:47:36.459 DEBUG:teuthology.misc:Transferring archived files from smithi132:/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/phlogistonjohn-2024-04-25_11:39:08-orch:cephadm-wip-phlogistonjohn-testing-2024-04-24-1356-distro-default-smithi/7673419/remote/smithi132/crash 2024-04-25T20:47:36.460 DEBUG:teuthology.orchestra.run.smithi132:> sudo tar c -f - -C /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/crash -- . 2024-04-25T20:47:36.499 INFO:teuthology.orchestra.run.smithi132.stderr:tar: /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-25T20:47:36.499 INFO:teuthology.orchestra.run.smithi132.stderr:tar: Error is not recoverable: exiting now 2024-04-25T20:47:36.501 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/phlogistonjohn-2024-04-25_11:39:08-orch:cephadm-wip-phlogistonjohn-testing-2024-04-24-1356-distro-default-smithi/7673419/remote/smithi150/crash 2024-04-25T20:47:36.502 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/crash -- . 2024-04-25T20:47:36.513 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/3114c848-033d-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-25T20:47:36.513 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2024-04-25T20:47:36.514 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-25T20:47:36.515 DEBUG:teuthology.orchestra.run.smithi132:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/3114c848-033d-11ef-bc93-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-25T20:47:36.558 INFO:teuthology.orchestra.run.smithi132.stdout:2024-04-25T20:14:46.957989+0000 mon.a (mon.0) 446 : cluster [WRN] Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2024-04-25T20:47:36.559 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-04-25T20:47:36.559 INFO:tasks.cephadm:Compressing logs... 2024-04-25T20:47:36.559 DEBUG:teuthology.orchestra.run.smithi132:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T20:47:36.601 DEBUG:teuthology.orchestra.run.smithi150:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T20:47:36.610 INFO:teuthology.orchestra.run.smithi132.stderr:find: '/var/log/rbd-target-api': No such file or directory 2024-04-25T20:47:36.613 INFO:teuthology.orchestra.run.smithi150.stderr:find: '/var/log/rbd-target-api': No such file or directory 2024-04-25T20:47:36.620 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-25T20:47:36.621 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/3114c848-033d-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-25T20:47:36.621 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/3114c848-033d-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi132.wpuwhp.log 2024-04-25T20:47:36.621 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/3114c848-033d-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/3114c848-033d-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-25T20:47:36.622 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose/var/log/ceph/3114c848-033d-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi132.wpuwhp.log: -- /var/log/ceph/3114c848-033d-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-25T20:47:36.622 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/3114c848-033d-11ef-bc93-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/3114c848-033d-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi132.wyahux.log 2024-04-25T20:47:36.624 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/3114c848-033d-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/3114c848-033d-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-25T20:47:36.624 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/3114c848-033d-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi132.wyahux.log: gzip -5 --verbose -- /var/log/ceph/3114c848-033d-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-25T20:47:36.624 INFO:teuthology.orchestra.run.smithi132.stderr: 84.9% -- replaced with /var/log/ceph/3114c848-033d-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-25T20:47:36.624 INFO:teuthology.orchestra.run.smithi132.stderr: 86.2% -- replaced with /var/log/ceph/3114c848-033d-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi132.wpuwhp.log.gz 2024-04-25T20:47:36.624 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/3114c848-033d-11ef-bc93-c7b262605968/ceph-osd.1.log: 81.8% -- replaced with /var/log/ceph/3114c848-033d-11ef-bc93-c7b262605968/ceph-mds.cephfs.smithi132.wyahux.log.gz 2024-04-25T20:47:36.624 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/3114c848-033d-11ef-bc93-c7b262605968/ceph.log 2024-04-25T20:47:36.624 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/3114c848-033d-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/3114c848-033d-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-25T20:47:36.625 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/3114c848-033d-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/3114c848-033d-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-25T20:47:36.628 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/3114c848-033d-11ef-bc93-c7b262605968/ceph-volume.log: /var/log/ceph/3114c848-033d-11ef-bc93-c7b262605968/ceph.audit.log: 90.4% -- replaced with /var/log/ceph/3114c848-033d-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-25T20:47:36.629 INFO:teuthology.orchestra.run.smithi132.stderr: 93.0% -- replaced with /var/log/ceph/3114c848-033d-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-25T20:47:36.632 INFO:teuthology.orchestra.run.smithi132.stderr: 94.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-25T20:47:36.633 INFO:teuthology.orchestra.run.smithi132.stderr: 91.3% -- replaced with /var/log/ceph/3114c848-033d-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-25T20:47:36.677 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-25T20:47:36.679 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/cephadm.log: 76.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-25T20:47:36.680 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-04-25T20:47:36.680 INFO:teuthology.orchestra.run.smithi150.stderr:real 0m0.074s 2024-04-25T20:47:36.680 INFO:teuthology.orchestra.run.smithi150.stderr:user 0m0.007s 2024-04-25T20:47:36.680 INFO:teuthology.orchestra.run.smithi150.stderr:sys 0m0.013s 2024-04-25T20:47:36.937 INFO:teuthology.orchestra.run.smithi132.stderr: 91.1% -- replaced with /var/log/ceph/3114c848-033d-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-25T20:47:37.524 INFO:teuthology.orchestra.run.smithi132.stderr: 89.4% -- replaced with /var/log/ceph/3114c848-033d-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-25T20:47:38.293 INFO:teuthology.orchestra.run.smithi132.stderr: 94.4% -- replaced with /var/log/ceph/3114c848-033d-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-25T20:47:38.446 INFO:teuthology.orchestra.run.smithi132.stderr: 94.3% -- replaced with /var/log/ceph/3114c848-033d-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-25T20:47:38.448 INFO:teuthology.orchestra.run.smithi132.stderr: 2024-04-25T20:47:38.448 INFO:teuthology.orchestra.run.smithi132.stderr:real 0m1.844s 2024-04-25T20:47:38.449 INFO:teuthology.orchestra.run.smithi132.stderr:user 0m4.563s 2024-04-25T20:47:38.449 INFO:teuthology.orchestra.run.smithi132.stderr:sys 0m0.206s 2024-04-25T20:47:38.449 INFO:tasks.cephadm:Archiving logs... 2024-04-25T20:47:38.450 DEBUG:teuthology.misc:Transferring archived files from smithi132:/var/log/ceph to /home/teuthworker/archive/phlogistonjohn-2024-04-25_11:39:08-orch:cephadm-wip-phlogistonjohn-testing-2024-04-24-1356-distro-default-smithi/7673419/remote/smithi132/log 2024-04-25T20:47:38.451 DEBUG:teuthology.orchestra.run.smithi132:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T20:47:39.131 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/log/ceph to /home/teuthworker/archive/phlogistonjohn-2024-04-25_11:39:08-orch:cephadm-wip-phlogistonjohn-testing-2024-04-24-1356-distro-default-smithi/7673419/remote/smithi150/log 2024-04-25T20:47:39.132 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T20:47:39.147 INFO:tasks.cephadm:Removing cluster... 2024-04-25T20:47:39.147 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3114c848-033d-11ef-bc93-c7b262605968 --force 2024-04-25T20:47:39.316 INFO:teuthology.orchestra.run.smithi132.stdout:Deleting cluster with fsid: 3114c848-033d-11ef-bc93-c7b262605968 2024-04-25T20:47:40.450 INFO:tasks.cephadm:Removing cephadm ... 2024-04-25T20:47:40.454 DEBUG:teuthology.orchestra.run.smithi132:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-25T20:47:40.459 DEBUG:teuthology.orchestra.run.smithi150:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-25T20:47:40.465 INFO:tasks.cephadm:Teardown complete 2024-04-25T20:47:40.465 DEBUG:teuthology.run_tasks:Unwinding manager cephadm.deploy_samba_ad_dc 2024-04-25T20:47:40.479 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker stop samba-ad 2024-04-25T20:47:41.194 INFO:teuthology.orchestra.run.smithi150.stdout:samba-ad 2024-04-25T20:47:41.197 DEBUG:teuthology.orchestra.run.smithi150:> sudo docker rm samba-ad 2024-04-25T20:47:41.393 INFO:teuthology.orchestra.run.smithi150.stdout:samba-ad 2024-04-25T20:47:41.395 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -rf /var/lib/samba/container/logs /var/lib/samba/container/data 2024-04-25T20:47:41.408 INFO:tasks.cephadm:Resetting systemd-resolved state on smithi150 2024-04-25T20:47:41.408 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T20:47:41.408 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/systemd/resolved.conf 2024-04-25T20:47:41.460 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl restart systemd-resolved 2024-04-25T20:47:41.593 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-25T20:47:41.603 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-25T20:47:41.603 DEBUG:teuthology.orchestra.run.smithi132:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T20:47:41.606 DEBUG:teuthology.orchestra.run.smithi150:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T20:47:41.616 INFO:teuthology.orchestra.run.smithi132.stdout: remote refid st t when poll reach delay offset jitter 2024-04-25T20:47:41.617 INFO:teuthology.orchestra.run.smithi132.stdout:============================================================================== 2024-04-25T20:47:41.617 INFO:teuthology.orchestra.run.smithi132.stdout:*hv01.front.sepi 67.205.162.81 3 u 29 128 377 0.198 -4.824 0.359 2024-04-25T20:47:41.617 INFO:teuthology.orchestra.run.smithi132.stdout:+hv02.front.sepi 63.231.80.2 3 u 106 128 377 0.181 -2.685 0.331 2024-04-25T20:47:41.617 INFO:teuthology.orchestra.run.smithi132.stdout:+hv03.front.sepi 74.6.168.72 3 u 25 128 377 0.171 -5.096 0.278 2024-04-25T20:47:41.617 INFO:teuthology.orchestra.run.smithi132.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-04-25T20:47:41.622 INFO:teuthology.orchestra.run.smithi150.stdout: remote refid st t when poll reach delay offset jitter 2024-04-25T20:47:41.623 INFO:teuthology.orchestra.run.smithi150.stdout:============================================================================== 2024-04-25T20:47:41.623 INFO:teuthology.orchestra.run.smithi150.stdout:*hv01.front.sepi 67.205.162.81 3 u 49 64 377 0.130 -1.662 0.683 2024-04-25T20:47:41.623 INFO:teuthology.orchestra.run.smithi150.stdout:+hv02.front.sepi 63.231.80.2 3 u 36 64 377 0.071 -0.723 0.469 2024-04-25T20:47:41.623 INFO:teuthology.orchestra.run.smithi150.stdout:+hv03.front.sepi 74.6.168.72 3 u 35 64 377 0.098 -2.976 0.613 2024-04-25T20:47:41.623 INFO:teuthology.orchestra.run.smithi150.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-04-25T20:47:41.623 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-25T20:47:41.632 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-25T20:47:41.633 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-25T20:47:41.642 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-25T20:47:41.651 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-25T20:47:41.660 INFO:teuthology.task.internal:Duration was 3892.650346 seconds 2024-04-25T20:47:41.660 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-25T20:47:41.668 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-25T20:47:41.668 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T20:47:41.671 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T20:47:41.708 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-25T20:47:41.708 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi132.front.sepia.ceph.com 2024-04-25T20:47:41.708 DEBUG:teuthology.orchestra.run.smithi132:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T20:47:41.761 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi150.front.sepia.ceph.com 2024-04-25T20:47:41.761 DEBUG:teuthology.orchestra.run.smithi150:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T20:47:41.771 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-25T20:47:41.772 DEBUG:teuthology.orchestra.run.smithi132:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T20:47:41.805 DEBUG:teuthology.orchestra.run.smithi150:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T20:47:41.904 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-25T20:47:41.905 DEBUG:teuthology.orchestra.run.smithi132:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T20:47:41.907 DEBUG:teuthology.orchestra.run.smithi150:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T20:47:42.327 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-25T20:47:42.337 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-25T20:47:42.337 DEBUG:teuthology.orchestra.run.smithi132:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T20:47:42.349 DEBUG:teuthology.orchestra.run.smithi150:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T20:47:42.363 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-25T20:47:42.373 DEBUG:teuthology.orchestra.run.smithi132:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T20:47:42.393 DEBUG:teuthology.orchestra.run.smithi150:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T20:47:42.400 INFO:teuthology.orchestra.run.smithi132.stdout:kernel.core_pattern = core 2024-04-25T20:47:42.416 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = core 2024-04-25T20:47:42.442 DEBUG:teuthology.orchestra.run.smithi132:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T20:47:42.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T20:47:42.467 DEBUG:teuthology.orchestra.run.smithi150:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T20:47:42.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T20:47:42.489 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-25T20:47:42.499 INFO:teuthology.task.internal:Transferring archived files... 2024-04-25T20:47:42.500 DEBUG:teuthology.misc:Transferring archived files from smithi132:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/phlogistonjohn-2024-04-25_11:39:08-orch:cephadm-wip-phlogistonjohn-testing-2024-04-24-1356-distro-default-smithi/7673419/remote/smithi132 2024-04-25T20:47:42.500 DEBUG:teuthology.orchestra.run.smithi132:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T20:47:42.548 DEBUG:teuthology.misc:Transferring archived files from smithi150:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/phlogistonjohn-2024-04-25_11:39:08-orch:cephadm-wip-phlogistonjohn-testing-2024-04-24-1356-distro-default-smithi/7673419/remote/smithi150 2024-04-25T20:47:42.548 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T20:47:42.569 INFO:teuthology.task.internal:Removing archive directory... 2024-04-25T20:47:42.569 DEBUG:teuthology.orchestra.run.smithi132:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T20:47:42.585 DEBUG:teuthology.orchestra.run.smithi150:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T20:47:42.614 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-25T20:47:42.625 INFO:teuthology.task.internal:Not uploading archives. 2024-04-25T20:47:42.625 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-25T20:47:42.635 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-25T20:47:42.635 DEBUG:teuthology.orchestra.run.smithi132:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T20:47:42.637 DEBUG:teuthology.orchestra.run.smithi150:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T20:47:42.642 INFO:teuthology.orchestra.run.smithi132.stdout: 658468 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 20:47 /home/ubuntu/cephtest 2024-04-25T20:47:42.665 INFO:teuthology.orchestra.run.smithi150.stdout: 658712 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 20:47 /home/ubuntu/cephtest 2024-04-25T20:47:42.668 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-25T20:47:42.692 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-25T20:47:42.758 INFO:teuthology.nuke:Checking targets against current locks 2024-04-25T20:47:42.800 DEBUG:teuthology.nuke:shortname: smithi132 2024-04-25T20:47:42.800 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T20:47:42.829 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi132.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/phlogistonjohn-2024-04-25_11:39:08-orch:cephadm-wip-phlogistonjohn-testing-2024-04-24-1356-distro-default-smithi/7673419', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 19:32:55.755509', 'locked_by': 'scheduled_phlogistonjohn@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGj9DBZof78QeX9azISYEVgx344AwuMrtGc0Fgm+apJWdf45t4JMXmdAhhvUlDlH3BvqnqiRyNJR+0QXAxjW15w='} 2024-04-25T20:47:42.833 DEBUG:teuthology.nuke:shortname: smithi150 2024-04-25T20:47:42.833 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T20:47:42.862 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/phlogistonjohn-2024-04-25_11:39:08-orch:cephadm-wip-phlogistonjohn-testing-2024-04-24-1356-distro-default-smithi/7673419', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 19:32:55.757167', 'locked_by': 'scheduled_phlogistonjohn@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOJKSXfi3X07tQKGC42atfFsxAAdAfPbrIyzwQX4GLc6HAKdWy5vbDCvhkVmn7vI9cOf/PKSATmgJ4kwgn7PZ88='} 2024-04-25T20:47:42.890 INFO:teuthology.orchestra.console.smithi132:Power off 2024-04-25T20:47:42.891 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-25T20:47:42.916 DEBUG:teuthology.orchestra.console.smithi132:power off output: Chassis Power Control: Down/Off 2024-04-25T20:47:42.916 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T20:47:42.932 INFO:teuthology.orchestra.console.smithi150:Power off 2024-04-25T20:47:42.932 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-25T20:47:42.945 DEBUG:teuthology.orchestra.console.smithi132:check power output: Chassis Power is on 2024-04-25T20:47:42.983 DEBUG:teuthology.orchestra.console.smithi150:power off output: Chassis Power Control: Down/Off 2024-04-25T20:47:42.983 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T20:47:43.016 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on 2024-04-25T20:47:46.947 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T20:47:47.017 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T20:47:47.058 DEBUG:teuthology.orchestra.console.smithi132:check power output: Chassis Power is on 2024-04-25T20:47:47.128 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on 2024-04-25T20:47:51.063 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T20:47:51.129 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T20:47:51.175 DEBUG:teuthology.orchestra.console.smithi132:check power output: Chassis Power is off 2024-04-25T20:47:51.239 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on 2024-04-25T20:47:51.275 INFO:teuthology.orchestra.console.smithi132:Power off completed 2024-04-25T20:47:55.242 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T20:47:55.353 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is off 2024-04-25T20:47:55.454 INFO:teuthology.orchestra.console.smithi150:Power off completed 2024-04-25T20:47:55.558 INFO:teuthology.run:Summary data: description: orch:cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_mgr_res_dom} duration: 3892.6503455638885 failure_reason: reached maximum tries (301) after waiting for 300 seconds owner: scheduled_phlogistonjohn@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=bfb1dfe28a1741d19b1a00c3492edbc9 status: fail success: false 2024-04-25T20:47:55.559 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T20:47:55.636 INFO:teuthology.run:FAIL