2024-04-24T17:13:29.200 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-24T17:13:29.201 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/adking-2024-04-24_15:49:18-orch:cephadm-wip-adk-testing-2024-04-24-0937-distro-default-smithi/7671850 --owner scheduled_adking@teuthology --description orch:cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} --verbose --name adking-2024-04-24_15:49:18-orch:cephadm-wip-adk-testing-2024-04-24-0937-distro-default-smithi -- /home/teuthworker/archive/adking-2024-04-24_15:49:18-orch:cephadm-wip-adk-testing-2024-04-24-0937-distro-default-smithi/7671850/orig.config.yaml 2024-04-24T17:13:29.230 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-24T17:13:29.316 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/adking-2024-04-24_15:49:18-orch:cephadm-wip-adk-testing-2024-04-24-0937-distro-default-smithi/7671850 branch: wip-adk-testing-2024-04-24-0937 description: orch:cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} email: adking@redhat.com first_in_suite: false job_id: '7671850' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: adking-2024-04-24_15:49:18-orch:cephadm-wip-adk-testing-2024-04-24-0937-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-adk-testing-2024-04-24-0937 ceph: conf: client: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms bind msgr2: false ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mgrc: 20 debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon warn on pool no app: false paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd class default list: '*' osd class load list: '*' osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 crush_tunables: jewel flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - reached quota - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) log-only-match: - CEPHADM_ mon_bind_msgr2: false sha1: 5904d29475f5be602879d9fb26280e89b808d5cc ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: 5904d29475f5be602879d9fb26280e89b808d5cc workunit: branch: wip-adk-testing-2024-04-24-0937 sha1: 5904d29475f5be602879d9fb26280e89b808d5cc owner: scheduled_adking@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 2532 sha1: 5904d29475f5be602879d9fb26280e89b808d5cc sleep_before_teardown: 0 subset: 1/16 suite: orch:cephadm suite_branch: wip-adk-testing-2024-04-24-0937 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_5904d29475f5be602879d9fb26280e89b808d5cc/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 5904d29475f5be602879d9fb26280e89b808d5cc targets: smithi032.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJhPwZXiDgl9ZAcEizV9fY7iWJ/ltclp1IPZAitkGKbEdcOM0LSRMmFZruXmmTE9TxQifb3wjzmfKGPW1fHn2fk= smithi082.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKpPIX8xrVbszAlxPyVcXR2kbxZ9olpvCuOj9FIZgFXyNL+DZ3Cy2N8pApUd0OYdmL4z6aEcIt5v9lxIXM6PipU= tasks: - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-24_15:49:18 tube: smithi user: adking verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-24T17:13:29.316 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_5904d29475f5be602879d9fb26280e89b808d5cc/qa; will attempt to use it 2024-04-24T17:13:29.317 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_5904d29475f5be602879d9fb26280e89b808d5cc/qa/tasks 2024-04-24T17:13:29.317 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-24T17:13:29.318 INFO:teuthology.task.internal:Checking packages... 2024-04-24T17:13:29.341 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '5904d29475f5be602879d9fb26280e89b808d5cc' 2024-04-24T17:13:29.341 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-24T17:13:29.342 INFO:teuthology.packaging:ref: None 2024-04-24T17:13:29.342 INFO:teuthology.packaging:tag: None 2024-04-24T17:13:29.342 INFO:teuthology.packaging:branch: wip-adk-testing-2024-04-24-0937 2024-04-24T17:13:29.342 INFO:teuthology.packaging:sha1: 5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T17:13:29.342 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-adk-testing-2024-04-24-0937 2024-04-24T17:13:29.563 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3285-g5904d294-1jammy 2024-04-24T17:13:29.564 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-24T17:13:29.574 INFO:teuthology.task.internal:no buildpackages task found 2024-04-24T17:13:29.574 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-24T17:13:29.622 INFO:teuthology.task.internal:Saving configuration 2024-04-24T17:13:29.634 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-24T17:13:29.656 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-24T17:13:29.683 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi032.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-04-24_15:49:18-orch:cephadm-wip-adk-testing-2024-04-24-0937-distro-default-smithi/7671850', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 17:08:05.312896', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILyuRMSUZt8BMsUSVE4QZU4OIPQq61WzcYfNf/ohQlcn'} 2024-04-24T17:13:29.711 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi082.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-04-24_15:49:18-orch:cephadm-wip-adk-testing-2024-04-24-0937-distro-default-smithi/7671850', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 17:08:05.314199', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILXfQJkk9OWalZri0teLeYN+KFf3rfgTF7MPq7OxuNhB'} 2024-04-24T17:13:29.712 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-24T17:13:29.730 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-04-24T17:13:29.730 INFO:teuthology.task.internal:roles: ubuntu@smithi082.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-04-24T17:13:29.730 INFO:teuthology.run_tasks:Running task console_log... 2024-04-24T17:13:29.850 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fca87ea84c0>, signals=[15]) 2024-04-24T17:13:29.851 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-24T17:13:29.860 INFO:teuthology.task.internal:Opening connections... 2024-04-24T17:13:29.860 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi032.front.sepia.ceph.com 2024-04-24T17:13:29.862 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T17:13:29.944 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi082.front.sepia.ceph.com 2024-04-24T17:13:29.945 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T17:13:30.025 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-24T17:13:30.033 DEBUG:teuthology.orchestra.run.smithi032:> uname -m 2024-04-24T17:13:30.039 INFO:teuthology.orchestra.run.smithi032.stdout:x86_64 2024-04-24T17:13:30.039 DEBUG:teuthology.orchestra.run.smithi032:> cat /etc/os-release 2024-04-24T17:13:30.087 INFO:teuthology.orchestra.run.smithi032.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-24T17:13:30.087 INFO:teuthology.orchestra.run.smithi032.stdout:NAME="Ubuntu" 2024-04-24T17:13:30.087 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION_ID="22.04" 2024-04-24T17:13:30.087 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-24T17:13:30.087 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION_CODENAME=jammy 2024-04-24T17:13:30.087 INFO:teuthology.orchestra.run.smithi032.stdout:ID=ubuntu 2024-04-24T17:13:30.087 INFO:teuthology.orchestra.run.smithi032.stdout:ID_LIKE=debian 2024-04-24T17:13:30.087 INFO:teuthology.orchestra.run.smithi032.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-24T17:13:30.087 INFO:teuthology.orchestra.run.smithi032.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-24T17:13:30.087 INFO:teuthology.orchestra.run.smithi032.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-24T17:13:30.087 INFO:teuthology.orchestra.run.smithi032.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-24T17:13:30.087 INFO:teuthology.orchestra.run.smithi032.stdout:UBUNTU_CODENAME=jammy 2024-04-24T17:13:30.088 INFO:teuthology.lock.ops:Updating smithi032.front.sepia.ceph.com on lock server 2024-04-24T17:13:30.110 DEBUG:teuthology.orchestra.run.smithi082:> uname -m 2024-04-24T17:13:30.116 INFO:teuthology.orchestra.run.smithi082.stdout:x86_64 2024-04-24T17:13:30.116 DEBUG:teuthology.orchestra.run.smithi082:> cat /etc/os-release 2024-04-24T17:13:30.161 INFO:teuthology.orchestra.run.smithi082.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-24T17:13:30.161 INFO:teuthology.orchestra.run.smithi082.stdout:NAME="Ubuntu" 2024-04-24T17:13:30.161 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION_ID="22.04" 2024-04-24T17:13:30.161 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-24T17:13:30.162 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION_CODENAME=jammy 2024-04-24T17:13:30.162 INFO:teuthology.orchestra.run.smithi082.stdout:ID=ubuntu 2024-04-24T17:13:30.162 INFO:teuthology.orchestra.run.smithi082.stdout:ID_LIKE=debian 2024-04-24T17:13:30.162 INFO:teuthology.orchestra.run.smithi082.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-24T17:13:30.162 INFO:teuthology.orchestra.run.smithi082.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-24T17:13:30.162 INFO:teuthology.orchestra.run.smithi082.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-24T17:13:30.162 INFO:teuthology.orchestra.run.smithi082.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-24T17:13:30.162 INFO:teuthology.orchestra.run.smithi082.stdout:UBUNTU_CODENAME=jammy 2024-04-24T17:13:30.162 INFO:teuthology.lock.ops:Updating smithi082.front.sepia.ceph.com on lock server 2024-04-24T17:13:30.176 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-24T17:13:30.186 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-24T17:13:30.214 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-24T17:13:30.214 DEBUG:teuthology.orchestra.run.smithi032:> test '!' -e /home/ubuntu/cephtest 2024-04-24T17:13:30.216 DEBUG:teuthology.orchestra.run.smithi082:> test '!' -e /home/ubuntu/cephtest 2024-04-24T17:13:30.221 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-24T17:13:30.247 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-24T17:13:30.247 DEBUG:teuthology.orchestra.run.smithi032:> test -z $(ls -A /var/lib/ceph) 2024-04-24T17:13:30.264 DEBUG:teuthology.orchestra.run.smithi082:> test -z $(ls -A /var/lib/ceph) 2024-04-24T17:13:30.273 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-24T17:13:30.323 INFO:teuthology.run_tasks:Running task kernel... 2024-04-24T17:13:30.340 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-24T17:13:30.341 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-24T17:13:30.341 DEBUG:teuthology.orchestra.run.smithi032:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-24T17:13:30.341 DEBUG:teuthology.orchestra.run.smithi082:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-24T17:13:30.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T17:13:30.346 DEBUG:teuthology.orchestra.run.smithi032:> uname -r 2024-04-24T17:13:30.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T17:13:30.347 DEBUG:teuthology.orchestra.run.smithi082:> uname -r 2024-04-24T17:13:30.393 INFO:teuthology.orchestra.run.smithi082.stdout:5.15.0-56-generic 2024-04-24T17:13:30.394 INFO:teuthology.task.kernel:Running kernel on smithi082: 5.15.0-56-generic 2024-04-24T17:13:30.394 DEBUG:teuthology.orchestra.run.smithi082:> sudo apt-get clean 2024-04-24T17:13:30.395 INFO:teuthology.orchestra.run.smithi032.stdout:5.15.0-56-generic 2024-04-24T17:13:30.395 INFO:teuthology.task.kernel:Running kernel on smithi032: 5.15.0-56-generic 2024-04-24T17:13:30.395 DEBUG:teuthology.orchestra.run.smithi032:> sudo apt-get clean 2024-04-24T17:13:30.530 DEBUG:teuthology.orchestra.run.smithi082:> sudo apt-get update 2024-04-24T17:13:30.531 DEBUG:teuthology.orchestra.run.smithi032:> sudo apt-get update 2024-04-24T17:13:30.694 INFO:teuthology.orchestra.run.smithi082.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-24T17:13:30.705 INFO:teuthology.orchestra.run.smithi032.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-24T17:13:30.707 INFO:teuthology.orchestra.run.smithi032.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-24T17:13:30.708 INFO:teuthology.orchestra.run.smithi032.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-24T17:13:30.790 INFO:teuthology.orchestra.run.smithi032.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-24T17:13:30.806 INFO:teuthology.orchestra.run.smithi082.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-24T17:13:30.890 INFO:teuthology.orchestra.run.smithi082.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-24T17:13:30.924 INFO:teuthology.orchestra.run.smithi082.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,392 kB] 2024-04-24T17:13:30.968 INFO:teuthology.orchestra.run.smithi032.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [452 kB] 2024-04-24T17:13:31.040 INFO:teuthology.orchestra.run.smithi082.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [452 kB] 2024-04-24T17:13:31.045 INFO:teuthology.orchestra.run.smithi082.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-04-24T17:13:31.049 INFO:teuthology.orchestra.run.smithi082.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-24T17:13:31.049 INFO:teuthology.orchestra.run.smithi082.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,768 kB] 2024-04-24T17:13:31.052 INFO:teuthology.orchestra.run.smithi032.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,610 kB] 2024-04-24T17:13:31.053 INFO:teuthology.orchestra.run.smithi032.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,392 kB] 2024-04-24T17:13:31.075 INFO:teuthology.orchestra.run.smithi082.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-24T17:13:31.079 INFO:teuthology.orchestra.run.smithi082.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [299 kB] 2024-04-24T17:13:31.082 INFO:teuthology.orchestra.run.smithi082.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-24T17:13:31.082 INFO:teuthology.orchestra.run.smithi082.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [601 kB] 2024-04-24T17:13:31.089 INFO:teuthology.orchestra.run.smithi082.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-24T17:13:31.099 INFO:teuthology.orchestra.run.smithi082.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-24T17:13:31.100 INFO:teuthology.orchestra.run.smithi082.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-24T17:13:31.103 INFO:teuthology.orchestra.run.smithi082.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-24T17:13:31.104 INFO:teuthology.orchestra.run.smithi082.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-24T17:13:31.107 INFO:teuthology.orchestra.run.smithi032.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-04-24T17:13:31.112 INFO:teuthology.orchestra.run.smithi032.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-24T17:13:31.112 INFO:teuthology.orchestra.run.smithi032.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-24T17:13:31.113 INFO:teuthology.orchestra.run.smithi032.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,768 kB] 2024-04-24T17:13:31.125 INFO:teuthology.orchestra.run.smithi082.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-24T17:13:31.139 INFO:teuthology.orchestra.run.smithi032.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [299 kB] 2024-04-24T17:13:31.142 INFO:teuthology.orchestra.run.smithi032.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-24T17:13:31.143 INFO:teuthology.orchestra.run.smithi032.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-24T17:13:31.146 INFO:teuthology.orchestra.run.smithi082.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-24T17:13:31.152 INFO:teuthology.orchestra.run.smithi032.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [601 kB] 2024-04-24T17:13:31.160 INFO:teuthology.orchestra.run.smithi032.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-24T17:13:31.161 INFO:teuthology.orchestra.run.smithi032.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-24T17:13:31.162 INFO:teuthology.orchestra.run.smithi032.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-24T17:13:31.162 INFO:teuthology.orchestra.run.smithi032.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-24T17:13:31.198 INFO:teuthology.orchestra.run.smithi032.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [618 kB] 2024-04-24T17:13:31.198 INFO:teuthology.orchestra.run.smithi032.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-24T17:13:31.199 INFO:teuthology.orchestra.run.smithi032.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-24T17:13:31.204 INFO:teuthology.orchestra.run.smithi032.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB] 2024-04-24T17:13:31.206 INFO:teuthology.orchestra.run.smithi032.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-24T17:13:31.206 INFO:teuthology.orchestra.run.smithi032.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,830 kB] 2024-04-24T17:13:31.230 INFO:teuthology.orchestra.run.smithi032.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-24T17:13:31.230 INFO:teuthology.orchestra.run.smithi032.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [311 kB] 2024-04-24T17:13:31.233 INFO:teuthology.orchestra.run.smithi032.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-24T17:13:31.233 INFO:teuthology.orchestra.run.smithi032.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,070 kB] 2024-04-24T17:13:31.246 INFO:teuthology.orchestra.run.smithi082.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-24T17:13:31.246 INFO:teuthology.orchestra.run.smithi032.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [700 kB] 2024-04-24T17:13:31.252 INFO:teuthology.orchestra.run.smithi032.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-24T17:13:31.255 INFO:teuthology.orchestra.run.smithi032.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-24T17:13:31.256 INFO:teuthology.orchestra.run.smithi032.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-24T17:13:31.256 INFO:teuthology.orchestra.run.smithi032.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-24T17:13:31.275 INFO:teuthology.orchestra.run.smithi032.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-24T17:13:31.275 INFO:teuthology.orchestra.run.smithi032.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-24T17:13:31.276 INFO:teuthology.orchestra.run.smithi032.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [82.9 kB] 2024-04-24T17:13:31.277 INFO:teuthology.orchestra.run.smithi032.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [90.8 kB] 2024-04-24T17:13:31.278 INFO:teuthology.orchestra.run.smithi032.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-24T17:13:31.278 INFO:teuthology.orchestra.run.smithi032.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-24T17:13:31.278 INFO:teuthology.orchestra.run.smithi032.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [29.8 kB] 2024-04-24T17:13:31.278 INFO:teuthology.orchestra.run.smithi032.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [18.5 kB] 2024-04-24T17:13:31.298 INFO:teuthology.orchestra.run.smithi032.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-24T17:13:31.298 INFO:teuthology.orchestra.run.smithi032.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-24T17:13:31.390 INFO:teuthology.orchestra.run.smithi082.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [618 kB] 2024-04-24T17:13:31.610 INFO:teuthology.orchestra.run.smithi082.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,610 kB] 2024-04-24T17:13:31.813 INFO:teuthology.orchestra.run.smithi082.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB] 2024-04-24T17:13:31.829 INFO:teuthology.orchestra.run.smithi082.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-24T17:13:31.829 INFO:teuthology.orchestra.run.smithi082.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-24T17:13:31.831 INFO:teuthology.orchestra.run.smithi082.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,830 kB] 2024-04-24T17:13:31.902 INFO:teuthology.orchestra.run.smithi082.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [311 kB] 2024-04-24T17:13:31.909 INFO:teuthology.orchestra.run.smithi082.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-24T17:13:32.158 INFO:teuthology.orchestra.run.smithi082.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [700 kB] 2024-04-24T17:13:32.842 INFO:teuthology.orchestra.run.smithi082.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,070 kB] 2024-04-24T17:13:33.047 INFO:teuthology.orchestra.run.smithi082.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-24T17:13:33.080 INFO:teuthology.orchestra.run.smithi082.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-24T17:13:33.081 INFO:teuthology.orchestra.run.smithi082.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-24T17:13:33.082 INFO:teuthology.orchestra.run.smithi082.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-24T17:13:33.095 INFO:teuthology.orchestra.run.smithi082.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-24T17:13:33.095 INFO:teuthology.orchestra.run.smithi082.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-24T17:13:33.096 INFO:teuthology.orchestra.run.smithi082.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [82.9 kB] 2024-04-24T17:13:33.102 INFO:teuthology.orchestra.run.smithi082.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [90.8 kB] 2024-04-24T17:13:33.116 INFO:teuthology.orchestra.run.smithi082.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-24T17:13:33.287 INFO:teuthology.orchestra.run.smithi082.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-24T17:13:33.371 INFO:teuthology.orchestra.run.smithi082.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [29.8 kB] 2024-04-24T17:13:33.483 INFO:teuthology.orchestra.run.smithi082.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [18.5 kB] 2024-04-24T17:13:33.541 INFO:teuthology.orchestra.run.smithi082.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-24T17:13:33.573 INFO:teuthology.orchestra.run.smithi082.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-24T17:13:37.099 INFO:teuthology.orchestra.run.smithi032.stdout:Fetched 13.3 MB in 2s (6,697 kB/s) 2024-04-24T17:13:37.875 INFO:teuthology.orchestra.run.smithi082.stdout:Fetched 13.3 MB in 3s (4,482 kB/s) 2024-04-24T17:13:38.675 INFO:teuthology.orchestra.run.smithi032.stdout:Reading package lists... 2024-04-24T17:13:38.694 DEBUG:teuthology.orchestra.run.smithi032:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-24T17:13:38.756 INFO:teuthology.orchestra.run.smithi032.stdout:Reading package lists... 2024-04-24T17:13:38.960 INFO:teuthology.orchestra.run.smithi032.stdout:Building dependency tree... 2024-04-24T17:13:38.960 INFO:teuthology.orchestra.run.smithi032.stdout:Reading state information... 2024-04-24T17:13:39.142 INFO:teuthology.orchestra.run.smithi032.stdout:The following packages were automatically installed and are no longer required: 2024-04-24T17:13:39.142 INFO:teuthology.orchestra.run.smithi032.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-24T17:13:39.142 INFO:teuthology.orchestra.run.smithi032.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-24T17:13:39.143 INFO:teuthology.orchestra.run.smithi032.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-24T17:13:39.143 INFO:teuthology.orchestra.run.smithi032.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-24T17:13:39.143 INFO:teuthology.orchestra.run.smithi032.stdout: python2.7-minimal 2024-04-24T17:13:39.143 INFO:teuthology.orchestra.run.smithi032.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-24T17:13:39.144 INFO:teuthology.orchestra.run.smithi032.stdout:The following additional packages will be installed: 2024-04-24T17:13:39.144 INFO:teuthology.orchestra.run.smithi032.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-24T17:13:39.144 INFO:teuthology.orchestra.run.smithi032.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-24T17:13:39.145 INFO:teuthology.orchestra.run.smithi032.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-24T17:13:39.146 INFO:teuthology.orchestra.run.smithi032.stdout:Suggested packages: 2024-04-24T17:13:39.146 INFO:teuthology.orchestra.run.smithi032.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-24T17:13:39.146 INFO:teuthology.orchestra.run.smithi032.stdout:Recommended packages: 2024-04-24T17:13:39.146 INFO:teuthology.orchestra.run.smithi032.stdout: thermald 2024-04-24T17:13:39.203 INFO:teuthology.orchestra.run.smithi032.stdout:The following NEW packages will be installed: 2024-04-24T17:13:39.203 INFO:teuthology.orchestra.run.smithi032.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-24T17:13:39.203 INFO:teuthology.orchestra.run.smithi032.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-24T17:13:39.204 INFO:teuthology.orchestra.run.smithi032.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-24T17:13:39.205 INFO:teuthology.orchestra.run.smithi032.stdout:The following packages will be upgraded: 2024-04-24T17:13:39.205 INFO:teuthology.orchestra.run.smithi032.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-24T17:13:39.237 INFO:teuthology.orchestra.run.smithi082.stdout:Reading package lists... 2024-04-24T17:13:39.255 DEBUG:teuthology.orchestra.run.smithi082:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-24T17:13:39.320 INFO:teuthology.orchestra.run.smithi082.stdout:Reading package lists... 2024-04-24T17:13:39.387 INFO:teuthology.orchestra.run.smithi032.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-24T17:13:39.387 INFO:teuthology.orchestra.run.smithi032.stdout:Need to get 113 MB of archives. 2024-04-24T17:13:39.387 INFO:teuthology.orchestra.run.smithi032.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-24T17:13:39.387 INFO:teuthology.orchestra.run.smithi032.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-24T17:13:39.520 INFO:teuthology.orchestra.run.smithi082.stdout:Building dependency tree... 2024-04-24T17:13:39.520 INFO:teuthology.orchestra.run.smithi082.stdout:Reading state information... 2024-04-24T17:13:39.697 INFO:teuthology.orchestra.run.smithi082.stdout:The following packages were automatically installed and are no longer required: 2024-04-24T17:13:39.697 INFO:teuthology.orchestra.run.smithi082.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-24T17:13:39.697 INFO:teuthology.orchestra.run.smithi082.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-24T17:13:39.698 INFO:teuthology.orchestra.run.smithi082.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-24T17:13:39.698 INFO:teuthology.orchestra.run.smithi082.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-24T17:13:39.698 INFO:teuthology.orchestra.run.smithi082.stdout: python2.7-minimal 2024-04-24T17:13:39.698 INFO:teuthology.orchestra.run.smithi082.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-24T17:13:39.699 INFO:teuthology.orchestra.run.smithi082.stdout:The following additional packages will be installed: 2024-04-24T17:13:39.699 INFO:teuthology.orchestra.run.smithi082.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-24T17:13:39.699 INFO:teuthology.orchestra.run.smithi082.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-24T17:13:39.700 INFO:teuthology.orchestra.run.smithi082.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-24T17:13:39.701 INFO:teuthology.orchestra.run.smithi082.stdout:Suggested packages: 2024-04-24T17:13:39.701 INFO:teuthology.orchestra.run.smithi082.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-24T17:13:39.701 INFO:teuthology.orchestra.run.smithi082.stdout:Recommended packages: 2024-04-24T17:13:39.701 INFO:teuthology.orchestra.run.smithi082.stdout: thermald 2024-04-24T17:13:39.754 INFO:teuthology.orchestra.run.smithi082.stdout:The following NEW packages will be installed: 2024-04-24T17:13:39.754 INFO:teuthology.orchestra.run.smithi082.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-24T17:13:39.754 INFO:teuthology.orchestra.run.smithi082.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-24T17:13:39.755 INFO:teuthology.orchestra.run.smithi082.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-24T17:13:39.756 INFO:teuthology.orchestra.run.smithi082.stdout:The following packages will be upgraded: 2024-04-24T17:13:39.757 INFO:teuthology.orchestra.run.smithi082.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-24T17:13:39.937 INFO:teuthology.orchestra.run.smithi082.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-24T17:13:39.938 INFO:teuthology.orchestra.run.smithi082.stdout:Need to get 113 MB of archives. 2024-04-24T17:13:39.938 INFO:teuthology.orchestra.run.smithi082.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-24T17:13:39.938 INFO:teuthology.orchestra.run.smithi082.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-24T17:13:40.827 INFO:teuthology.orchestra.run.smithi032.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-24T17:13:41.163 INFO:teuthology.orchestra.run.smithi032.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-24T17:13:41.395 INFO:teuthology.orchestra.run.smithi082.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-24T17:13:41.742 INFO:teuthology.orchestra.run.smithi082.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-24T17:13:43.314 INFO:teuthology.orchestra.run.smithi032.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-24T17:13:43.314 INFO:teuthology.orchestra.run.smithi032.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-24T17:13:43.315 INFO:teuthology.orchestra.run.smithi032.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-24T17:13:43.889 INFO:teuthology.orchestra.run.smithi082.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-24T17:13:43.889 INFO:teuthology.orchestra.run.smithi082.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-24T17:13:43.889 INFO:teuthology.orchestra.run.smithi082.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-24T17:13:44.376 INFO:teuthology.orchestra.run.smithi032.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-24T17:13:44.386 INFO:teuthology.orchestra.run.smithi082.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-24T17:13:44.485 INFO:teuthology.orchestra.run.smithi032.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-24T17:13:44.506 INFO:teuthology.orchestra.run.smithi082.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-24T17:13:44.916 INFO:teuthology.orchestra.run.smithi082.stdout:Fetched 113 MB in 5s (23.9 MB/s) 2024-04-24T17:13:44.974 INFO:teuthology.orchestra.run.smithi032.stdout:Fetched 113 MB in 5s (21.5 MB/s) 2024-04-24T17:13:45.092 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-24T17:13:45.170 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-24T17:13:47.068 INFO:teuthology.orchestra.run.smithi082.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-24T17:13:47.072 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-24T17:13:47.122 INFO:teuthology.orchestra.run.smithi032.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-24T17:13:47.126 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-24T17:13:47.164 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T17:13:47.257 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T17:13:49.341 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-24T17:13:49.362 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-24T17:13:49.469 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T17:13:49.521 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-24T17:13:49.542 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-24T17:13:49.636 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T17:13:49.919 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-24T17:13:49.940 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-24T17:13:49.994 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T17:13:50.099 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-24T17:13:50.120 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-24T17:13:50.157 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T17:13:56.671 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-24T17:13:56.790 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-24T17:13:57.056 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-24T17:13:57.110 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-24T17:13:57.177 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-24T17:13:57.213 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-24T17:13:57.576 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-24T17:13:57.700 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-24T17:13:57.707 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-24T17:13:57.730 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-24T17:13:57.804 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-24T17:13:58.011 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-24T17:13:58.022 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-24T17:13:58.061 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-24T17:14:03.377 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-24T17:14:03.413 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-24T17:14:03.442 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T17:14:04.182 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-24T17:14:04.216 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-24T17:14:04.253 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T17:14:06.124 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-24T17:14:07.013 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-24T17:14:07.363 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-24T17:14:07.509 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T17:14:07.635 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-24T17:14:07.772 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T17:14:07.796 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-24T17:14:07.930 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-24T17:14:08.313 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-24T17:14:08.434 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T17:14:08.561 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-24T17:14:08.714 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T17:14:09.122 INFO:teuthology.orchestra.run.smithi032.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-24T17:14:09.122 INFO:teuthology.orchestra.run.smithi032.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-24T17:14:09.122 INFO:teuthology.orchestra.run.smithi032.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-24T17:14:09.122 INFO:teuthology.orchestra.run.smithi032.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-24T17:14:09.282 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T17:14:10.051 INFO:teuthology.orchestra.run.smithi082.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-24T17:14:10.051 INFO:teuthology.orchestra.run.smithi082.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-24T17:14:10.051 INFO:teuthology.orchestra.run.smithi082.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-24T17:14:10.051 INFO:teuthology.orchestra.run.smithi082.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-24T17:14:10.199 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T17:14:10.759 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T17:14:11.617 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T17:14:12.894 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-24T17:14:13.019 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-24T17:14:13.168 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-24T17:14:13.294 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-24T17:14:13.412 INFO:teuthology.orchestra.run.smithi082.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T17:14:13.469 INFO:teuthology.orchestra.run.smithi082.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-24T17:14:13.469 INFO:teuthology.orchestra.run.smithi082.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-24T17:14:13.704 INFO:teuthology.orchestra.run.smithi032.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T17:14:13.759 INFO:teuthology.orchestra.run.smithi032.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-24T17:14:13.759 INFO:teuthology.orchestra.run.smithi032.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-24T17:14:27.808 INFO:teuthology.orchestra.run.smithi082.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-24T17:14:27.808 INFO:teuthology.orchestra.run.smithi082.stdout:Sourcing file `/etc/default/grub' 2024-04-24T17:14:27.832 INFO:teuthology.orchestra.run.smithi082.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-24T17:14:27.856 INFO:teuthology.orchestra.run.smithi082.stdout:Generating grub configuration file ... 2024-04-24T17:14:28.116 INFO:teuthology.orchestra.run.smithi032.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-24T17:14:28.116 INFO:teuthology.orchestra.run.smithi032.stdout:Sourcing file `/etc/default/grub' 2024-04-24T17:14:28.123 INFO:teuthology.orchestra.run.smithi032.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-24T17:14:28.142 INFO:teuthology.orchestra.run.smithi032.stdout:Generating grub configuration file ... 2024-04-24T17:14:28.176 INFO:teuthology.orchestra.run.smithi082.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-24T17:14:28.193 INFO:teuthology.orchestra.run.smithi082.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-24T17:14:28.435 INFO:teuthology.orchestra.run.smithi032.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-24T17:14:28.449 INFO:teuthology.orchestra.run.smithi032.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-24T17:14:28.511 INFO:teuthology.orchestra.run.smithi082.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-24T17:14:28.516 INFO:teuthology.orchestra.run.smithi082.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-24T17:14:28.580 INFO:teuthology.orchestra.run.smithi082.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-24T17:14:28.585 INFO:teuthology.orchestra.run.smithi082.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-24T17:14:28.736 INFO:teuthology.orchestra.run.smithi082.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-24T17:14:28.737 INFO:teuthology.orchestra.run.smithi082.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-24T17:14:28.737 INFO:teuthology.orchestra.run.smithi082.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-24T17:14:28.777 INFO:teuthology.orchestra.run.smithi082.stdout:done 2024-04-24T17:14:28.780 INFO:teuthology.orchestra.run.smithi032.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-24T17:14:28.786 INFO:teuthology.orchestra.run.smithi032.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-24T17:14:28.846 INFO:teuthology.orchestra.run.smithi032.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-24T17:14:28.852 INFO:teuthology.orchestra.run.smithi032.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-24T17:14:29.024 INFO:teuthology.orchestra.run.smithi032.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-24T17:14:29.024 INFO:teuthology.orchestra.run.smithi032.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-24T17:14:29.024 INFO:teuthology.orchestra.run.smithi032.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-24T17:14:29.070 INFO:teuthology.orchestra.run.smithi032.stdout:done 2024-04-24T17:14:29.265 DEBUG:teuthology.orchestra.run.smithi082:> dpkg -s linux-image-generic 2024-04-24T17:14:29.287 INFO:teuthology.orchestra.run.smithi082.stdout:Package: linux-image-generic 2024-04-24T17:14:29.287 INFO:teuthology.orchestra.run.smithi082.stdout:Status: install ok installed 2024-04-24T17:14:29.287 INFO:teuthology.orchestra.run.smithi082.stdout:Priority: optional 2024-04-24T17:14:29.287 INFO:teuthology.orchestra.run.smithi082.stdout:Section: kernel 2024-04-24T17:14:29.288 INFO:teuthology.orchestra.run.smithi082.stdout:Installed-Size: 21 2024-04-24T17:14:29.288 INFO:teuthology.orchestra.run.smithi082.stdout:Maintainer: Ubuntu Kernel Team 2024-04-24T17:14:29.288 INFO:teuthology.orchestra.run.smithi082.stdout:Architecture: amd64 2024-04-24T17:14:29.288 INFO:teuthology.orchestra.run.smithi082.stdout:Source: linux-meta 2024-04-24T17:14:29.288 INFO:teuthology.orchestra.run.smithi082.stdout:Version: 5.15.0.105.102 2024-04-24T17:14:29.288 INFO:teuthology.orchestra.run.smithi082.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-24T17:14:29.288 INFO:teuthology.orchestra.run.smithi082.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-24T17:14:29.288 INFO:teuthology.orchestra.run.smithi082.stdout:Recommends: thermald 2024-04-24T17:14:29.288 INFO:teuthology.orchestra.run.smithi082.stdout:Description: Generic Linux kernel image 2024-04-24T17:14:29.288 INFO:teuthology.orchestra.run.smithi082.stdout: This package will always depend on the latest generic kernel image 2024-04-24T17:14:29.288 INFO:teuthology.orchestra.run.smithi082.stdout: available. 2024-04-24T17:14:29.288 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-24T17:14:29.289 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-24T17:14:29.289 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-24T17:14:29.289 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-24T17:14:29.289 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi082.front.sepia.ceph.com, path=None, version=distro) 2024-04-24T17:14:29.289 DEBUG:teuthology.orchestra.run.smithi082:> sudo apt-get clean 2024-04-24T17:14:29.456 DEBUG:teuthology.orchestra.run.smithi082:> sudo apt-get update 2024-04-24T17:14:29.517 DEBUG:teuthology.orchestra.run.smithi032:> dpkg -s linux-image-generic 2024-04-24T17:14:29.541 INFO:teuthology.orchestra.run.smithi032.stdout:Package: linux-image-generic 2024-04-24T17:14:29.541 INFO:teuthology.orchestra.run.smithi032.stdout:Status: install ok installed 2024-04-24T17:14:29.541 INFO:teuthology.orchestra.run.smithi032.stdout:Priority: optional 2024-04-24T17:14:29.541 INFO:teuthology.orchestra.run.smithi032.stdout:Section: kernel 2024-04-24T17:14:29.541 INFO:teuthology.orchestra.run.smithi032.stdout:Installed-Size: 21 2024-04-24T17:14:29.541 INFO:teuthology.orchestra.run.smithi032.stdout:Maintainer: Ubuntu Kernel Team 2024-04-24T17:14:29.541 INFO:teuthology.orchestra.run.smithi032.stdout:Architecture: amd64 2024-04-24T17:14:29.541 INFO:teuthology.orchestra.run.smithi032.stdout:Source: linux-meta 2024-04-24T17:14:29.541 INFO:teuthology.orchestra.run.smithi032.stdout:Version: 5.15.0.105.102 2024-04-24T17:14:29.541 INFO:teuthology.orchestra.run.smithi032.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-24T17:14:29.541 INFO:teuthology.orchestra.run.smithi032.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-24T17:14:29.542 INFO:teuthology.orchestra.run.smithi032.stdout:Recommends: thermald 2024-04-24T17:14:29.542 INFO:teuthology.orchestra.run.smithi032.stdout:Description: Generic Linux kernel image 2024-04-24T17:14:29.542 INFO:teuthology.orchestra.run.smithi032.stdout: This package will always depend on the latest generic kernel image 2024-04-24T17:14:29.542 INFO:teuthology.orchestra.run.smithi032.stdout: available. 2024-04-24T17:14:29.542 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-24T17:14:29.542 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-24T17:14:29.542 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-24T17:14:29.542 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-24T17:14:29.542 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi032.front.sepia.ceph.com, path=None, version=distro) 2024-04-24T17:14:29.542 DEBUG:teuthology.orchestra.run.smithi032:> sudo apt-get clean 2024-04-24T17:14:29.609 INFO:teuthology.orchestra.run.smithi082.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-24T17:14:29.609 INFO:teuthology.orchestra.run.smithi082.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-24T17:14:29.643 INFO:teuthology.orchestra.run.smithi082.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-24T17:14:29.683 INFO:teuthology.orchestra.run.smithi082.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-24T17:14:29.757 DEBUG:teuthology.orchestra.run.smithi032:> sudo apt-get update 2024-04-24T17:14:29.900 INFO:teuthology.orchestra.run.smithi032.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-24T17:14:29.900 INFO:teuthology.orchestra.run.smithi032.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-24T17:14:29.909 INFO:teuthology.orchestra.run.smithi032.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-24T17:14:29.944 INFO:teuthology.orchestra.run.smithi032.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-24T17:14:30.938 INFO:teuthology.orchestra.run.smithi082.stdout:Reading package lists... 2024-04-24T17:14:30.955 DEBUG:teuthology.orchestra.run.smithi082:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-24T17:14:31.017 INFO:teuthology.orchestra.run.smithi082.stdout:Reading package lists... 2024-04-24T17:14:31.225 INFO:teuthology.orchestra.run.smithi082.stdout:Building dependency tree... 2024-04-24T17:14:31.225 INFO:teuthology.orchestra.run.smithi082.stdout:Reading state information... 2024-04-24T17:14:31.317 INFO:teuthology.orchestra.run.smithi032.stdout:Reading package lists... 2024-04-24T17:14:31.336 DEBUG:teuthology.orchestra.run.smithi032:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-24T17:14:31.389 INFO:teuthology.orchestra.run.smithi032.stdout:Reading package lists... 2024-04-24T17:14:31.401 INFO:teuthology.orchestra.run.smithi082.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-24T17:14:31.401 INFO:teuthology.orchestra.run.smithi082.stdout:The following packages were automatically installed and are no longer required: 2024-04-24T17:14:31.401 INFO:teuthology.orchestra.run.smithi082.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-24T17:14:31.401 INFO:teuthology.orchestra.run.smithi082.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-24T17:14:31.401 INFO:teuthology.orchestra.run.smithi082.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-24T17:14:31.402 INFO:teuthology.orchestra.run.smithi082.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-24T17:14:31.402 INFO:teuthology.orchestra.run.smithi082.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-24T17:14:31.402 INFO:teuthology.orchestra.run.smithi082.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-24T17:14:31.444 INFO:teuthology.orchestra.run.smithi082.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-24T17:14:31.446 DEBUG:teuthology.orchestra.run.smithi082:> dpkg -s linux-image-generic 2024-04-24T17:14:31.462 INFO:teuthology.orchestra.run.smithi082.stdout:Package: linux-image-generic 2024-04-24T17:14:31.462 INFO:teuthology.orchestra.run.smithi082.stdout:Status: install ok installed 2024-04-24T17:14:31.462 INFO:teuthology.orchestra.run.smithi082.stdout:Priority: optional 2024-04-24T17:14:31.462 INFO:teuthology.orchestra.run.smithi082.stdout:Section: kernel 2024-04-24T17:14:31.462 INFO:teuthology.orchestra.run.smithi082.stdout:Installed-Size: 21 2024-04-24T17:14:31.462 INFO:teuthology.orchestra.run.smithi082.stdout:Maintainer: Ubuntu Kernel Team 2024-04-24T17:14:31.462 INFO:teuthology.orchestra.run.smithi082.stdout:Architecture: amd64 2024-04-24T17:14:31.463 INFO:teuthology.orchestra.run.smithi082.stdout:Source: linux-meta 2024-04-24T17:14:31.463 INFO:teuthology.orchestra.run.smithi082.stdout:Version: 5.15.0.105.102 2024-04-24T17:14:31.463 INFO:teuthology.orchestra.run.smithi082.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-24T17:14:31.463 INFO:teuthology.orchestra.run.smithi082.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-24T17:14:31.463 INFO:teuthology.orchestra.run.smithi082.stdout:Recommends: thermald 2024-04-24T17:14:31.463 INFO:teuthology.orchestra.run.smithi082.stdout:Description: Generic Linux kernel image 2024-04-24T17:14:31.463 INFO:teuthology.orchestra.run.smithi082.stdout: This package will always depend on the latest generic kernel image 2024-04-24T17:14:31.463 INFO:teuthology.orchestra.run.smithi082.stdout: available. 2024-04-24T17:14:31.463 DEBUG:teuthology.orchestra.run.smithi082:> mktemp 2024-04-24T17:14:31.508 INFO:teuthology.orchestra.run.smithi082.stdout:/tmp/tmp.vqMruzop7a 2024-04-24T17:14:31.509 DEBUG:teuthology.orchestra.run.smithi082:> sudo cp /boot/grub/grub.cfg /tmp/tmp.vqMruzop7a 2024-04-24T17:14:31.571 INFO:teuthology.orchestra.run.smithi032.stdout:Building dependency tree... 2024-04-24T17:14:31.572 INFO:teuthology.orchestra.run.smithi032.stdout:Reading state information... 2024-04-24T17:14:31.573 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 0666 /tmp/tmp.vqMruzop7a 2024-04-24T17:14:31.692 DEBUG:teuthology.orchestra.remote:smithi082:/tmp/tmp.vqMruzop7a is 10KB 2024-04-24T17:14:31.705 DEBUG:teuthology.orchestra.run.smithi082:> rm -fr /tmp/tmp.vqMruzop7a 2024-04-24T17:14:31.711 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-24T17:14:31.735 INFO:teuthology.orchestra.run.smithi032.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-24T17:14:31.735 INFO:teuthology.orchestra.run.smithi032.stdout:The following packages were automatically installed and are no longer required: 2024-04-24T17:14:31.735 INFO:teuthology.orchestra.run.smithi032.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-24T17:14:31.735 INFO:teuthology.orchestra.run.smithi032.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-24T17:14:31.735 INFO:teuthology.orchestra.run.smithi032.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-24T17:14:31.736 INFO:teuthology.orchestra.run.smithi032.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-24T17:14:31.736 INFO:teuthology.orchestra.run.smithi032.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-24T17:14:31.736 INFO:teuthology.orchestra.run.smithi032.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-24T17:14:31.764 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-04-24T17:14:31.764 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-24T17:14:31.764 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-24T17:14:31.775 INFO:teuthology.orchestra.run.smithi032.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-24T17:14:31.777 DEBUG:teuthology.orchestra.run.smithi032:> dpkg -s linux-image-generic 2024-04-24T17:14:31.833 INFO:teuthology.orchestra.run.smithi032.stdout:Package: linux-image-generic 2024-04-24T17:14:31.833 INFO:teuthology.orchestra.run.smithi032.stdout:Status: install ok installed 2024-04-24T17:14:31.833 INFO:teuthology.orchestra.run.smithi032.stdout:Priority: optional 2024-04-24T17:14:31.833 INFO:teuthology.orchestra.run.smithi032.stdout:Section: kernel 2024-04-24T17:14:31.833 INFO:teuthology.orchestra.run.smithi032.stdout:Installed-Size: 21 2024-04-24T17:14:31.833 INFO:teuthology.orchestra.run.smithi032.stdout:Maintainer: Ubuntu Kernel Team 2024-04-24T17:14:31.833 INFO:teuthology.orchestra.run.smithi032.stdout:Architecture: amd64 2024-04-24T17:14:31.833 INFO:teuthology.orchestra.run.smithi032.stdout:Source: linux-meta 2024-04-24T17:14:31.834 INFO:teuthology.orchestra.run.smithi032.stdout:Version: 5.15.0.105.102 2024-04-24T17:14:31.834 INFO:teuthology.orchestra.run.smithi032.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-24T17:14:31.834 INFO:teuthology.orchestra.run.smithi032.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-24T17:14:31.834 INFO:teuthology.orchestra.run.smithi032.stdout:Recommends: thermald 2024-04-24T17:14:31.834 INFO:teuthology.orchestra.run.smithi032.stdout:Description: Generic Linux kernel image 2024-04-24T17:14:31.834 INFO:teuthology.orchestra.run.smithi032.stdout: This package will always depend on the latest generic kernel image 2024-04-24T17:14:31.834 INFO:teuthology.orchestra.run.smithi032.stdout: available. 2024-04-24T17:14:31.834 DEBUG:teuthology.orchestra.run.smithi032:> mktemp 2024-04-24T17:14:31.846 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-24T17:14:31.847 DEBUG:teuthology.orchestra.run.smithi082:> sudo update-grub 2024-04-24T17:14:31.878 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/tmp.0zZuqI5WC2 2024-04-24T17:14:31.878 DEBUG:teuthology.orchestra.run.smithi032:> sudo cp /boot/grub/grub.cfg /tmp/tmp.0zZuqI5WC2 2024-04-24T17:14:31.934 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0666 /tmp/tmp.0zZuqI5WC2 2024-04-24T17:14:32.047 DEBUG:teuthology.orchestra.remote:smithi032:/tmp/tmp.0zZuqI5WC2 is 10KB 2024-04-24T17:14:32.059 DEBUG:teuthology.orchestra.run.smithi032:> rm -fr /tmp/tmp.0zZuqI5WC2 2024-04-24T17:14:32.064 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-24T17:14:32.119 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-24T17:14:32.119 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-24T17:14:32.119 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-24T17:14:32.187 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-24T17:14:32.187 DEBUG:teuthology.orchestra.run.smithi032:> sudo update-grub 2024-04-24T17:14:33.136 INFO:teuthology.orchestra.run.smithi082.stderr:Sourcing file `/etc/default/grub' 2024-04-24T17:14:33.137 INFO:teuthology.orchestra.run.smithi082.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-24T17:14:33.140 INFO:teuthology.orchestra.run.smithi082.stderr:Generating grub configuration file ... 2024-04-24T17:14:33.322 INFO:teuthology.orchestra.run.smithi082.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-24T17:14:33.329 INFO:teuthology.orchestra.run.smithi082.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-24T17:14:33.518 INFO:teuthology.orchestra.run.smithi032.stderr:Sourcing file `/etc/default/grub' 2024-04-24T17:14:33.519 INFO:teuthology.orchestra.run.smithi032.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-24T17:14:33.522 INFO:teuthology.orchestra.run.smithi032.stderr:Generating grub configuration file ... 2024-04-24T17:14:33.548 INFO:teuthology.orchestra.run.smithi082.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-24T17:14:33.553 INFO:teuthology.orchestra.run.smithi082.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-24T17:14:33.595 INFO:teuthology.orchestra.run.smithi082.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-24T17:14:33.601 INFO:teuthology.orchestra.run.smithi082.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-24T17:14:33.709 INFO:teuthology.orchestra.run.smithi032.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-24T17:14:33.716 INFO:teuthology.orchestra.run.smithi032.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-24T17:14:33.729 INFO:teuthology.orchestra.run.smithi082.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-24T17:14:33.729 INFO:teuthology.orchestra.run.smithi082.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-24T17:14:33.730 INFO:teuthology.orchestra.run.smithi082.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-24T17:14:33.742 INFO:teuthology.orchestra.run.smithi082.stderr:done 2024-04-24T17:14:33.749 DEBUG:teuthology.orchestra.run.smithi082:> sudo shutdown -r now 2024-04-24T17:14:33.954 INFO:teuthology.orchestra.run.smithi032.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-24T17:14:33.959 INFO:teuthology.orchestra.run.smithi032.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-24T17:14:34.000 INFO:teuthology.orchestra.run.smithi032.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-24T17:14:34.005 INFO:teuthology.orchestra.run.smithi032.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-24T17:14:34.123 INFO:teuthology.orchestra.run.smithi032.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-24T17:14:34.123 INFO:teuthology.orchestra.run.smithi032.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-24T17:14:34.123 INFO:teuthology.orchestra.run.smithi032.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-24T17:14:34.150 INFO:teuthology.orchestra.run.smithi032.stderr:done 2024-04-24T17:14:34.163 DEBUG:teuthology.orchestra.run.smithi032:> sudo shutdown -r now 2024-04-24T17:15:03.782 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-24T17:15:03.783 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-04-24T17:15:03.784 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T17:15:04.166 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-24T17:15:04.166 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-04-24T17:15:04.167 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T17:15:22.270 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.82 2024-04-24T17:15:22.526 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.32 2024-04-24T17:15:31.278 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-04-24T17:15:31.279 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T17:15:31.527 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-04-24T17:15:31.528 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T17:15:34.338 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.82 2024-04-24T17:15:46.350 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-04-24T17:15:46.352 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T17:15:49.922 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.32 2024-04-24T17:16:01.934 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-04-24T17:16:01.935 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T17:16:04.990 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.32 2024-04-24T17:16:18.161 DEBUG:teuthology.orchestra.run.smithi082:> true 2024-04-24T17:16:18.720 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-04-24T17:16:18.720 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-105-generic"... 2024-04-24T17:16:18.720 DEBUG:teuthology.orchestra.run.smithi082:> uname -r 2024-04-24T17:16:18.766 INFO:teuthology.orchestra.run.smithi082.stdout:5.15.0-105-generic 2024-04-24T17:16:18.766 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-24T17:16:18.766 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-24T17:16:18.766 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-24T17:16:19.767 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-24T17:16:19.767 DEBUG:teuthology.orchestra.run.smithi082:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-24T17:16:19.991 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-04-24T17:16:19.991 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T17:16:20.129 INFO:teuthology.orchestra.run.smithi082.stdout:ttyS1 2024-04-24T17:16:20.159 DEBUG:teuthology.parallel:result is None 2024-04-24T17:16:20.218 DEBUG:teuthology.orchestra.run.smithi032:> true 2024-04-24T17:16:20.872 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-04-24T17:16:20.872 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-105-generic"... 2024-04-24T17:16:20.872 DEBUG:teuthology.orchestra.run.smithi032:> uname -r 2024-04-24T17:16:20.920 INFO:teuthology.orchestra.run.smithi032.stdout:5.15.0-105-generic 2024-04-24T17:16:20.920 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-24T17:16:20.920 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-24T17:16:20.920 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-24T17:16:21.921 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-24T17:16:21.921 DEBUG:teuthology.orchestra.run.smithi032:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-24T17:16:22.002 INFO:teuthology.orchestra.run.smithi032.stdout:ttyS1 2024-04-24T17:16:22.018 DEBUG:teuthology.parallel:result is None 2024-04-24T17:16:22.022 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-24T17:16:22.032 INFO:teuthology.task.internal:Creating test directory... 2024-04-24T17:16:22.032 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-24T17:16:22.035 DEBUG:teuthology.orchestra.run.smithi082:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-24T17:16:22.042 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-24T17:16:22.080 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-24T17:16:22.129 INFO:teuthology.task.internal:Creating archive directory... 2024-04-24T17:16:22.130 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-24T17:16:22.132 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-24T17:16:22.208 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-24T17:16:22.214 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-24T17:16:22.215 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-24T17:16:22.217 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-24T17:16:22.239 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T17:16:22.239 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T17:16:22.247 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T17:16:22.248 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T17:16:22.250 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-24T17:16:22.257 INFO:teuthology.task.internal:Configuring sudo... 2024-04-24T17:16:22.257 DEBUG:teuthology.orchestra.run.smithi032:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-24T17:16:22.293 DEBUG:teuthology.orchestra.run.smithi082:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-24T17:16:22.305 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-24T17:16:22.314 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-24T17:16:22.315 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-24T17:16:22.349 DEBUG:teuthology.orchestra.run.smithi082:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-24T17:16:22.355 DEBUG:teuthology.orchestra.run.smithi032:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T17:16:22.400 DEBUG:teuthology.orchestra.run.smithi032:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T17:16:22.448 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-24T17:16:22.448 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-24T17:16:22.509 DEBUG:teuthology.orchestra.run.smithi082:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T17:16:22.515 DEBUG:teuthology.orchestra.run.smithi082:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T17:16:22.562 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-04-24T17:16:22.562 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-24T17:16:22.638 DEBUG:teuthology.orchestra.run.smithi032:> sudo service rsyslog restart 2024-04-24T17:16:22.640 DEBUG:teuthology.orchestra.run.smithi082:> sudo service rsyslog restart 2024-04-24T17:16:22.710 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-24T17:16:22.718 INFO:teuthology.task.internal:Starting timer... 2024-04-24T17:16:22.718 INFO:teuthology.run_tasks:Running task pcp... 2024-04-24T17:16:22.767 INFO:teuthology.run_tasks:Running task selinux... 2024-04-24T17:16:22.816 DEBUG:teuthology.task.selinux:Excluding smithi032: OS 'ubuntu' does not support SELinux 2024-04-24T17:16:22.816 DEBUG:teuthology.task.selinux:Excluding smithi082: OS 'ubuntu' does not support SELinux 2024-04-24T17:16:22.816 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-24T17:16:22.817 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-04-24T17:16:22.817 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-24T17:16:22.817 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-24T17:16:22.865 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-24T17:16:22.882 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-24T17:16:22.883 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi032.front.sepia.ceph.com,smithi082.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-24T17:23:28.332 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi032.front.sepia.ceph.com'), Remote(name='ubuntu@smithi082.front.sepia.ceph.com')] 2024-04-24T17:23:28.333 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-04-24T17:23:28.334 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T17:23:28.421 DEBUG:teuthology.orchestra.run.smithi032:> true 2024-04-24T17:23:28.498 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-04-24T17:23:28.499 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-04-24T17:23:28.499 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T17:23:28.579 DEBUG:teuthology.orchestra.run.smithi082:> true 2024-04-24T17:23:28.654 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-04-24T17:23:28.654 INFO:teuthology.run_tasks:Running task clock... 2024-04-24T17:23:28.667 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-24T17:23:28.667 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-24T17:23:28.668 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T17:23:28.670 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-24T17:23:28.670 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T17:23:28.700 INFO:teuthology.orchestra.run.smithi032.stdout:24 Apr 17:23:28 ntpd[17102]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-24T17:23:28.700 INFO:teuthology.orchestra.run.smithi032.stdout:24 Apr 17:23:28 ntpd[17102]: Command line: ntpd -gq 2024-04-24T17:23:28.700 INFO:teuthology.orchestra.run.smithi032.stdout:24 Apr 17:23:28 ntpd[17102]: ---------------------------------------------------- 2024-04-24T17:23:28.701 INFO:teuthology.orchestra.run.smithi032.stdout:24 Apr 17:23:28 ntpd[17102]: ntp-4 is maintained by Network Time Foundation, 2024-04-24T17:23:28.701 INFO:teuthology.orchestra.run.smithi032.stdout:24 Apr 17:23:28 ntpd[17102]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-24T17:23:28.701 INFO:teuthology.orchestra.run.smithi032.stdout:24 Apr 17:23:28 ntpd[17102]: corporation. Support and training for ntp-4 are 2024-04-24T17:23:28.701 INFO:teuthology.orchestra.run.smithi032.stdout:24 Apr 17:23:28 ntpd[17102]: available at https://www.nwtime.org/support 2024-04-24T17:23:28.701 INFO:teuthology.orchestra.run.smithi032.stdout:24 Apr 17:23:28 ntpd[17102]: ---------------------------------------------------- 2024-04-24T17:23:28.702 INFO:teuthology.orchestra.run.smithi032.stdout:24 Apr 17:23:28 ntpd[17102]: proto: precision = 0.047 usec (-24) 2024-04-24T17:23:28.702 INFO:teuthology.orchestra.run.smithi032.stdout:24 Apr 17:23:28 ntpd[17102]: basedate set to 2022-02-04 2024-04-24T17:23:28.702 INFO:teuthology.orchestra.run.smithi032.stdout:24 Apr 17:23:28 ntpd[17102]: gps base set to 2022-02-06 (week 2196) 2024-04-24T17:23:28.702 INFO:teuthology.orchestra.run.smithi032.stdout:24 Apr 17:23:28 ntpd[17102]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-24T17:23:28.702 INFO:teuthology.orchestra.run.smithi032.stdout:24 Apr 17:23:28 ntpd[17102]: restrict ::: KOD does nothing without LIMITED. 2024-04-24T17:23:28.702 INFO:teuthology.orchestra.run.smithi032.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-24T17:23:28.702 INFO:teuthology.orchestra.run.smithi032.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-24T17:23:28.703 INFO:teuthology.orchestra.run.smithi032.stdout:24 Apr 17:23:28 ntpd[17102]: Listen and drop on 0 v6wildcard [::]:123 2024-04-24T17:23:28.703 INFO:teuthology.orchestra.run.smithi032.stdout:24 Apr 17:23:28 ntpd[17102]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-24T17:23:28.703 INFO:teuthology.orchestra.run.smithi032.stdout:24 Apr 17:23:28 ntpd[17102]: Listen normally on 2 lo 127.0.0.1:123 2024-04-24T17:23:28.703 INFO:teuthology.orchestra.run.smithi032.stdout:24 Apr 17:23:28 ntpd[17102]: Listen normally on 3 ens1f0 172.21.15.32:123 2024-04-24T17:23:28.703 INFO:teuthology.orchestra.run.smithi032.stdout:24 Apr 17:23:28 ntpd[17102]: Listen normally on 4 lo [::1]:123 2024-04-24T17:23:28.704 INFO:teuthology.orchestra.run.smithi032.stdout:24 Apr 17:23:28 ntpd[17102]: Listen normally on 5 ens1f0 [fe80::ae1f:6bff:fef8:2602%4]:123 2024-04-24T17:23:28.704 INFO:teuthology.orchestra.run.smithi032.stdout:24 Apr 17:23:28 ntpd[17102]: Listening on routing socket on fd #22 for interface updates 2024-04-24T17:23:28.729 INFO:teuthology.orchestra.run.smithi082.stdout:24 Apr 17:23:28 ntpd[17229]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-24T17:23:28.729 INFO:teuthology.orchestra.run.smithi082.stdout:24 Apr 17:23:28 ntpd[17229]: Command line: ntpd -gq 2024-04-24T17:23:28.730 INFO:teuthology.orchestra.run.smithi082.stdout:24 Apr 17:23:28 ntpd[17229]: ---------------------------------------------------- 2024-04-24T17:23:28.730 INFO:teuthology.orchestra.run.smithi082.stdout:24 Apr 17:23:28 ntpd[17229]: ntp-4 is maintained by Network Time Foundation, 2024-04-24T17:23:28.730 INFO:teuthology.orchestra.run.smithi082.stdout:24 Apr 17:23:28 ntpd[17229]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-24T17:23:28.730 INFO:teuthology.orchestra.run.smithi082.stdout:24 Apr 17:23:28 ntpd[17229]: corporation. Support and training for ntp-4 are 2024-04-24T17:23:28.730 INFO:teuthology.orchestra.run.smithi082.stdout:24 Apr 17:23:28 ntpd[17229]: available at https://www.nwtime.org/support 2024-04-24T17:23:28.730 INFO:teuthology.orchestra.run.smithi082.stdout:24 Apr 17:23:28 ntpd[17229]: ---------------------------------------------------- 2024-04-24T17:23:28.730 INFO:teuthology.orchestra.run.smithi082.stdout:24 Apr 17:23:28 ntpd[17229]: proto: precision = 0.068 usec (-24) 2024-04-24T17:23:28.730 INFO:teuthology.orchestra.run.smithi082.stdout:24 Apr 17:23:28 ntpd[17229]: basedate set to 2022-02-04 2024-04-24T17:23:28.730 INFO:teuthology.orchestra.run.smithi082.stdout:24 Apr 17:23:28 ntpd[17229]: gps base set to 2022-02-06 (week 2196) 2024-04-24T17:23:28.730 INFO:teuthology.orchestra.run.smithi082.stdout:24 Apr 17:23:28 ntpd[17229]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-24T17:23:28.731 INFO:teuthology.orchestra.run.smithi082.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-24T17:23:28.731 INFO:teuthology.orchestra.run.smithi082.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-24T17:23:28.732 INFO:teuthology.orchestra.run.smithi082.stdout:24 Apr 17:23:28 ntpd[17229]: restrict ::: KOD does nothing without LIMITED. 2024-04-24T17:23:28.732 INFO:teuthology.orchestra.run.smithi082.stdout:24 Apr 17:23:28 ntpd[17229]: Listen and drop on 0 v6wildcard [::]:123 2024-04-24T17:23:28.732 INFO:teuthology.orchestra.run.smithi082.stdout:24 Apr 17:23:28 ntpd[17229]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-24T17:23:28.732 INFO:teuthology.orchestra.run.smithi082.stdout:24 Apr 17:23:28 ntpd[17229]: Listen normally on 2 lo 127.0.0.1:123 2024-04-24T17:23:28.732 INFO:teuthology.orchestra.run.smithi082.stdout:24 Apr 17:23:28 ntpd[17229]: Listen normally on 3 enp3s0f1 172.21.15.82:123 2024-04-24T17:23:28.732 INFO:teuthology.orchestra.run.smithi082.stdout:24 Apr 17:23:28 ntpd[17229]: Listen normally on 4 lo [::1]:123 2024-04-24T17:23:28.732 INFO:teuthology.orchestra.run.smithi082.stdout:24 Apr 17:23:28 ntpd[17229]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d3ff%5]:123 2024-04-24T17:23:28.732 INFO:teuthology.orchestra.run.smithi082.stdout:24 Apr 17:23:28 ntpd[17229]: Listening on routing socket on fd #22 for interface updates 2024-04-24T17:23:29.701 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:29 ntpd[17102]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T17:23:29.701 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:29 ntpd[17102]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T17:23:29.701 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:29 ntpd[17102]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T17:23:29.701 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:29 ntpd[17102]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T17:23:29.701 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:29 ntpd[17102]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T17:23:29.702 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:29 ntpd[17102]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T17:23:29.729 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:29 ntpd[17229]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T17:23:29.729 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:29 ntpd[17229]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T17:23:29.729 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:29 ntpd[17229]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T17:23:29.729 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:29 ntpd[17229]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T17:23:29.730 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:29 ntpd[17229]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T17:23:29.730 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:29 ntpd[17229]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T17:23:30.701 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:30 ntpd[17102]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T17:23:30.701 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:30 ntpd[17102]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T17:23:30.701 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:30 ntpd[17102]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T17:23:30.702 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:30 ntpd[17102]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T17:23:30.702 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:30 ntpd[17102]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T17:23:30.702 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:30 ntpd[17102]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T17:23:31.701 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:31 ntpd[17102]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T17:23:31.702 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:31 ntpd[17102]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T17:23:31.702 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:31 ntpd[17102]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T17:23:31.702 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:31 ntpd[17102]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T17:23:31.702 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:31 ntpd[17102]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T17:23:31.702 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:31 ntpd[17102]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T17:23:31.702 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:31 ntpd[17102]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T17:23:31.702 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:31 ntpd[17102]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T17:23:31.729 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:31 ntpd[17229]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T17:23:31.730 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:31 ntpd[17229]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T17:23:31.730 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:31 ntpd[17229]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T17:23:31.730 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:31 ntpd[17229]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T17:23:31.730 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:31 ntpd[17229]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T17:23:31.730 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:31 ntpd[17229]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T17:23:32.702 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:32 ntpd[17102]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T17:23:32.702 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:32 ntpd[17102]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T17:23:32.702 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:32 ntpd[17102]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T17:23:32.702 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:32 ntpd[17102]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T17:23:32.702 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:32 ntpd[17102]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T17:23:32.702 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:32 ntpd[17102]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T17:23:32.702 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:32 ntpd[17102]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T17:23:32.703 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:32 ntpd[17102]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T17:23:32.703 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:32 ntpd[17102]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T17:23:32.703 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:32 ntpd[17102]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T17:23:32.703 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:32 ntpd[17102]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T17:23:32.703 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:32 ntpd[17102]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T17:23:32.729 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:32 ntpd[17229]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T17:23:32.729 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:32 ntpd[17229]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T17:23:32.730 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:32 ntpd[17229]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T17:23:32.730 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:32 ntpd[17229]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T17:23:32.730 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:32 ntpd[17229]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T17:23:32.730 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:32 ntpd[17229]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T17:23:32.730 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:32 ntpd[17229]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T17:23:32.730 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:32 ntpd[17229]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T17:23:33.701 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:33 ntpd[17102]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T17:23:33.701 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:33 ntpd[17102]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T17:23:33.702 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:33 ntpd[17102]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T17:23:33.702 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:33 ntpd[17102]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T17:23:33.702 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:33 ntpd[17102]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T17:23:33.702 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:33 ntpd[17102]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T17:23:33.730 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:33 ntpd[17229]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T17:23:33.730 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:33 ntpd[17229]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T17:23:33.730 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:33 ntpd[17229]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T17:23:33.730 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:33 ntpd[17229]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T17:23:33.730 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:33 ntpd[17229]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T17:23:33.730 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:33 ntpd[17229]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T17:23:33.730 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:33 ntpd[17229]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T17:23:33.731 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:33 ntpd[17229]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T17:23:33.731 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:33 ntpd[17229]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T17:23:33.731 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:33 ntpd[17229]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T17:23:33.731 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:33 ntpd[17229]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T17:23:33.731 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:33 ntpd[17229]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T17:23:34.701 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:34 ntpd[17102]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T17:23:34.702 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:34 ntpd[17102]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T17:23:34.702 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:34 ntpd[17102]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T17:23:34.702 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:34 ntpd[17102]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T17:23:34.702 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:34 ntpd[17102]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T17:23:34.702 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:34 ntpd[17102]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T17:23:34.702 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:34 ntpd[17102]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T17:23:34.702 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:34 ntpd[17102]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T17:23:34.702 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:34 ntpd[17102]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T17:23:34.702 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:34 ntpd[17102]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T17:23:34.702 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:34 ntpd[17102]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T17:23:34.702 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:34 ntpd[17102]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T17:23:34.729 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:34 ntpd[17229]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T17:23:34.729 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:34 ntpd[17229]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T17:23:34.729 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:34 ntpd[17229]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T17:23:34.730 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:34 ntpd[17229]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T17:23:34.730 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:34 ntpd[17229]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T17:23:34.730 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:34 ntpd[17229]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T17:23:35.701 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:35 ntpd[17102]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T17:23:35.701 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:35 ntpd[17102]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T17:23:35.701 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:35 ntpd[17102]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T17:23:35.702 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:35 ntpd[17102]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T17:23:35.702 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:35 ntpd[17102]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T17:23:35.702 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:35 ntpd[17102]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T17:23:35.702 INFO:teuthology.orchestra.run.smithi032.stdout:24 Apr 17:23:35 ntpd[17102]: ntpd: time slew -0.001173 s 2024-04-24T17:23:35.702 INFO:teuthology.orchestra.run.smithi032.stdout:ntpd: time slew -0.001173s 2024-04-24T17:23:35.703 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:35 ntpd[17102]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-24T17:23:35.703 INFO:teuthology.orchestra.run.smithi032.stderr:24 Apr 17:23:35 ntpd[17102]: can't open /var/log/ntpstats/loopstats.20240424: Permission denied 2024-04-24T17:23:35.730 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:35 ntpd[17229]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T17:23:35.730 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:35 ntpd[17229]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T17:23:35.730 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:35 ntpd[17229]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T17:23:35.730 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:35 ntpd[17229]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T17:23:35.730 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:35 ntpd[17229]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T17:23:35.730 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:35 ntpd[17229]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T17:23:35.730 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:35 ntpd[17229]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T17:23:35.730 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:35 ntpd[17229]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T17:23:35.731 INFO:teuthology.orchestra.run.smithi082.stdout:24 Apr 17:23:35 ntpd[17229]: ntpd: time slew -0.000952 s 2024-04-24T17:23:35.731 INFO:teuthology.orchestra.run.smithi082.stdout:ntpd: time slew -0.000952s 2024-04-24T17:23:35.731 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:35 ntpd[17229]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-24T17:23:35.731 INFO:teuthology.orchestra.run.smithi082.stderr:24 Apr 17:23:35 ntpd[17229]: can't open /var/log/ntpstats/loopstats.20240424: Permission denied 2024-04-24T17:23:35.750 INFO:teuthology.orchestra.run.smithi032.stdout: remote refid st t when poll reach delay offset jitter 2024-04-24T17:23:35.750 INFO:teuthology.orchestra.run.smithi032.stdout:============================================================================== 2024-04-24T17:23:35.750 INFO:teuthology.orchestra.run.smithi032.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-24T17:23:35.750 INFO:teuthology.orchestra.run.smithi032.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-24T17:23:35.750 INFO:teuthology.orchestra.run.smithi032.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-24T17:23:35.750 INFO:teuthology.orchestra.run.smithi032.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-24T17:23:35.782 INFO:teuthology.orchestra.run.smithi082.stdout: remote refid st t when poll reach delay offset jitter 2024-04-24T17:23:35.782 INFO:teuthology.orchestra.run.smithi082.stdout:============================================================================== 2024-04-24T17:23:35.782 INFO:teuthology.orchestra.run.smithi082.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-24T17:23:35.782 INFO:teuthology.orchestra.run.smithi082.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-24T17:23:35.783 INFO:teuthology.orchestra.run.smithi082.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-24T17:23:35.783 INFO:teuthology.orchestra.run.smithi082.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-24T17:23:35.783 INFO:teuthology.run_tasks:Running task install... 2024-04-24T17:23:35.792 DEBUG:teuthology.task.install:project ceph 2024-04-24T17:23:35.792 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5904d29475f5be602879d9fb26280e89b808d5cc'}} 2024-04-24T17:23:35.792 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5904d29475f5be602879d9fb26280e89b808d5cc'} 2024-04-24T17:23:35.792 INFO:teuthology.task.install:Using flavor: default 2024-04-24T17:23:35.798 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-24T17:23:35.798 INFO:teuthology.task.install:extra packages: [] 2024-04-24T17:23:35.798 DEBUG:teuthology.orchestra.run.smithi032:> sudo apt-key list | grep Ceph 2024-04-24T17:23:35.799 DEBUG:teuthology.orchestra.run.smithi082:> sudo apt-key list | grep Ceph 2024-04-24T17:23:35.858 INFO:teuthology.orchestra.run.smithi032.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-24T17:23:35.881 INFO:teuthology.orchestra.run.smithi082.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-24T17:23:35.886 INFO:teuthology.orchestra.run.smithi032.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-24T17:23:35.886 INFO:teuthology.orchestra.run.smithi032.stdout:uid [ unknown] Ceph.com (release key) 2024-04-24T17:23:35.887 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-24T17:23:35.888 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T17:23:35.907 INFO:teuthology.orchestra.run.smithi082.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-24T17:23:35.907 INFO:teuthology.orchestra.run.smithi082.stdout:uid [ unknown] Ceph.com (release key) 2024-04-24T17:23:35.907 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-24T17:23:35.907 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T17:23:36.123 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default/ 2024-04-24T17:23:36.123 INFO:teuthology.task.install.deb:Package version is 19.0.0-3285-g5904d294-1jammy 2024-04-24T17:23:36.130 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default/ 2024-04-24T17:23:36.130 INFO:teuthology.task.install.deb:Package version is 19.0.0-3285-g5904d294-1jammy 2024-04-24T17:23:36.234 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-24T17:23:36.234 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-24T17:23:36.240 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-04-24T17:23:36.240 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-24T17:23:36.249 DEBUG:teuthology.orchestra.run.smithi032:> sudo apt-get update 2024-04-24T17:23:36.254 DEBUG:teuthology.orchestra.run.smithi082:> sudo apt-get update 2024-04-24T17:23:36.425 INFO:teuthology.orchestra.run.smithi082.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-24T17:23:36.475 INFO:teuthology.orchestra.run.smithi082.stdout:Ign:2 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy InRelease 2024-04-24T17:23:36.496 INFO:teuthology.orchestra.run.smithi082.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy Release [6,978 B] 2024-04-24T17:23:36.518 INFO:teuthology.orchestra.run.smithi082.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-24T17:23:36.538 INFO:teuthology.orchestra.run.smithi032.stdout:Ign:1 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy InRelease 2024-04-24T17:23:36.540 INFO:teuthology.orchestra.run.smithi082.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-04-24T17:23:36.546 INFO:teuthology.orchestra.run.smithi082.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-24T17:23:36.552 INFO:teuthology.orchestra.run.smithi032.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-24T17:23:36.553 INFO:teuthology.orchestra.run.smithi032.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-24T17:23:36.567 INFO:teuthology.orchestra.run.smithi082.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,397 B] 2024-04-24T17:23:36.575 INFO:teuthology.orchestra.run.smithi032.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy Release [6,978 B] 2024-04-24T17:23:36.613 INFO:teuthology.orchestra.run.smithi032.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-24T17:23:36.625 INFO:teuthology.orchestra.run.smithi032.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-24T17:23:36.628 INFO:teuthology.orchestra.run.smithi082.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-24T17:23:36.649 INFO:teuthology.orchestra.run.smithi032.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,397 B] 2024-04-24T17:23:36.706 INFO:teuthology.orchestra.run.smithi082.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-24T17:23:36.707 INFO:teuthology.orchestra.run.smithi032.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-24T17:23:36.819 INFO:teuthology.orchestra.run.smithi032.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-04-24T17:23:36.838 INFO:teuthology.orchestra.run.smithi082.stdout:Fetched 28.5 kB in 0s (70.1 kB/s) 2024-04-24T17:23:36.945 INFO:teuthology.orchestra.run.smithi032.stdout:Fetched 28.5 kB in 1s (55.9 kB/s) 2024-04-24T17:23:37.748 INFO:teuthology.orchestra.run.smithi082.stdout:Reading package lists... 2024-04-24T17:23:37.764 DEBUG:teuthology.orchestra.run.smithi082:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-3285-g5904d294-1jammy cephadm=19.0.0-3285-g5904d294-1jammy ceph-mds=19.0.0-3285-g5904d294-1jammy ceph-mgr=19.0.0-3285-g5904d294-1jammy ceph-common=19.0.0-3285-g5904d294-1jammy ceph-fuse=19.0.0-3285-g5904d294-1jammy ceph-test=19.0.0-3285-g5904d294-1jammy ceph-volume=19.0.0-3285-g5904d294-1jammy radosgw=19.0.0-3285-g5904d294-1jammy python3-rados=19.0.0-3285-g5904d294-1jammy python3-rgw=19.0.0-3285-g5904d294-1jammy python3-cephfs=19.0.0-3285-g5904d294-1jammy python3-rbd=19.0.0-3285-g5904d294-1jammy libcephfs2=19.0.0-3285-g5904d294-1jammy libcephfs-dev=19.0.0-3285-g5904d294-1jammy librados2=19.0.0-3285-g5904d294-1jammy librbd1=19.0.0-3285-g5904d294-1jammy rbd-fuse=19.0.0-3285-g5904d294-1jammy 2024-04-24T17:23:37.825 INFO:teuthology.orchestra.run.smithi082.stdout:Reading package lists... 2024-04-24T17:23:37.922 INFO:teuthology.orchestra.run.smithi032.stdout:Reading package lists... 2024-04-24T17:23:37.938 DEBUG:teuthology.orchestra.run.smithi032:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-3285-g5904d294-1jammy cephadm=19.0.0-3285-g5904d294-1jammy ceph-mds=19.0.0-3285-g5904d294-1jammy ceph-mgr=19.0.0-3285-g5904d294-1jammy ceph-common=19.0.0-3285-g5904d294-1jammy ceph-fuse=19.0.0-3285-g5904d294-1jammy ceph-test=19.0.0-3285-g5904d294-1jammy ceph-volume=19.0.0-3285-g5904d294-1jammy radosgw=19.0.0-3285-g5904d294-1jammy python3-rados=19.0.0-3285-g5904d294-1jammy python3-rgw=19.0.0-3285-g5904d294-1jammy python3-cephfs=19.0.0-3285-g5904d294-1jammy python3-rbd=19.0.0-3285-g5904d294-1jammy libcephfs2=19.0.0-3285-g5904d294-1jammy libcephfs-dev=19.0.0-3285-g5904d294-1jammy librados2=19.0.0-3285-g5904d294-1jammy librbd1=19.0.0-3285-g5904d294-1jammy rbd-fuse=19.0.0-3285-g5904d294-1jammy 2024-04-24T17:23:38.002 INFO:teuthology.orchestra.run.smithi032.stdout:Reading package lists... 2024-04-24T17:23:38.023 INFO:teuthology.orchestra.run.smithi082.stdout:Building dependency tree... 2024-04-24T17:23:38.024 INFO:teuthology.orchestra.run.smithi082.stdout:Reading state information... 2024-04-24T17:23:38.213 INFO:teuthology.orchestra.run.smithi032.stdout:Building dependency tree... 2024-04-24T17:23:38.214 INFO:teuthology.orchestra.run.smithi032.stdout:Reading state information... 2024-04-24T17:23:38.218 INFO:teuthology.orchestra.run.smithi082.stdout:The following packages were automatically installed and are no longer required: 2024-04-24T17:23:38.218 INFO:teuthology.orchestra.run.smithi082.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-24T17:23:38.218 INFO:teuthology.orchestra.run.smithi082.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-24T17:23:38.218 INFO:teuthology.orchestra.run.smithi082.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-24T17:23:38.219 INFO:teuthology.orchestra.run.smithi082.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-24T17:23:38.219 INFO:teuthology.orchestra.run.smithi082.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-24T17:23:38.219 INFO:teuthology.orchestra.run.smithi082.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-24T17:23:38.220 INFO:teuthology.orchestra.run.smithi082.stdout:The following additional packages will be installed: 2024-04-24T17:23:38.220 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-24T17:23:38.220 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-24T17:23:38.221 INFO:teuthology.orchestra.run.smithi082.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-24T17:23:38.221 INFO:teuthology.orchestra.run.smithi082.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-24T17:23:38.221 INFO:teuthology.orchestra.run.smithi082.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-24T17:23:38.221 INFO:teuthology.orchestra.run.smithi082.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-24T17:23:38.221 INFO:teuthology.orchestra.run.smithi082.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-24T17:23:38.221 INFO:teuthology.orchestra.run.smithi082.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-24T17:23:38.221 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-24T17:23:38.222 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-24T17:23:38.222 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-24T17:23:38.222 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-24T17:23:38.222 INFO:teuthology.orchestra.run.smithi082.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-24T17:23:38.222 INFO:teuthology.orchestra.run.smithi082.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-24T17:23:38.222 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-24T17:23:38.222 INFO:teuthology.orchestra.run.smithi082.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-24T17:23:38.222 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-24T17:23:38.222 INFO:teuthology.orchestra.run.smithi082.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-24T17:23:38.222 INFO:teuthology.orchestra.run.smithi082.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-24T17:23:38.222 INFO:teuthology.orchestra.run.smithi082.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-24T17:23:38.222 INFO:teuthology.orchestra.run.smithi082.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-24T17:23:38.222 INFO:teuthology.orchestra.run.smithi082.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-24T17:23:38.222 INFO:teuthology.orchestra.run.smithi082.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-24T17:23:38.223 INFO:teuthology.orchestra.run.smithi082.stdout: xmlstarlet zip 2024-04-24T17:23:38.223 INFO:teuthology.orchestra.run.smithi082.stdout:Suggested packages: 2024-04-24T17:23:38.223 INFO:teuthology.orchestra.run.smithi082.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-24T17:23:38.223 INFO:teuthology.orchestra.run.smithi082.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-24T17:23:38.223 INFO:teuthology.orchestra.run.smithi082.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-24T17:23:38.223 INFO:teuthology.orchestra.run.smithi082.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-24T17:23:38.223 INFO:teuthology.orchestra.run.smithi082.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-24T17:23:38.223 INFO:teuthology.orchestra.run.smithi082.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-24T17:23:38.224 INFO:teuthology.orchestra.run.smithi082.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-24T17:23:38.224 INFO:teuthology.orchestra.run.smithi082.stdout:Recommended packages: 2024-04-24T17:23:38.224 INFO:teuthology.orchestra.run.smithi082.stdout: btrfs-tools 2024-04-24T17:23:38.322 INFO:teuthology.orchestra.run.smithi082.stdout:The following NEW packages will be installed: 2024-04-24T17:23:38.323 INFO:teuthology.orchestra.run.smithi082.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-24T17:23:38.323 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-24T17:23:38.323 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-24T17:23:38.323 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-24T17:23:38.323 INFO:teuthology.orchestra.run.smithi082.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-24T17:23:38.323 INFO:teuthology.orchestra.run.smithi082.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-24T17:23:38.324 INFO:teuthology.orchestra.run.smithi082.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-24T17:23:38.324 INFO:teuthology.orchestra.run.smithi082.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-24T17:23:38.324 INFO:teuthology.orchestra.run.smithi082.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-24T17:23:38.324 INFO:teuthology.orchestra.run.smithi082.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-24T17:23:38.324 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-24T17:23:38.324 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-24T17:23:38.324 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-24T17:23:38.324 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-24T17:23:38.324 INFO:teuthology.orchestra.run.smithi082.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-24T17:23:38.325 INFO:teuthology.orchestra.run.smithi082.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-24T17:23:38.325 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-24T17:23:38.325 INFO:teuthology.orchestra.run.smithi082.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-24T17:23:38.325 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-24T17:23:38.325 INFO:teuthology.orchestra.run.smithi082.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-24T17:23:38.325 INFO:teuthology.orchestra.run.smithi082.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-24T17:23:38.325 INFO:teuthology.orchestra.run.smithi082.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-24T17:23:38.325 INFO:teuthology.orchestra.run.smithi082.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-24T17:23:38.325 INFO:teuthology.orchestra.run.smithi082.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-24T17:23:38.325 INFO:teuthology.orchestra.run.smithi082.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-24T17:23:38.325 INFO:teuthology.orchestra.run.smithi082.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-24T17:23:38.428 INFO:teuthology.orchestra.run.smithi032.stdout:The following packages were automatically installed and are no longer required: 2024-04-24T17:23:38.428 INFO:teuthology.orchestra.run.smithi032.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-24T17:23:38.429 INFO:teuthology.orchestra.run.smithi032.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-24T17:23:38.429 INFO:teuthology.orchestra.run.smithi032.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-24T17:23:38.429 INFO:teuthology.orchestra.run.smithi032.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-24T17:23:38.429 INFO:teuthology.orchestra.run.smithi032.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-24T17:23:38.429 INFO:teuthology.orchestra.run.smithi032.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-24T17:23:38.430 INFO:teuthology.orchestra.run.smithi032.stdout:The following additional packages will be installed: 2024-04-24T17:23:38.430 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-24T17:23:38.430 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-24T17:23:38.430 INFO:teuthology.orchestra.run.smithi032.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-24T17:23:38.430 INFO:teuthology.orchestra.run.smithi032.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-24T17:23:38.432 INFO:teuthology.orchestra.run.smithi032.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-24T17:23:38.432 INFO:teuthology.orchestra.run.smithi032.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-24T17:23:38.432 INFO:teuthology.orchestra.run.smithi032.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-24T17:23:38.432 INFO:teuthology.orchestra.run.smithi032.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-24T17:23:38.432 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-24T17:23:38.432 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-24T17:23:38.432 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-24T17:23:38.432 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-24T17:23:38.432 INFO:teuthology.orchestra.run.smithi032.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-24T17:23:38.432 INFO:teuthology.orchestra.run.smithi032.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-24T17:23:38.432 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-24T17:23:38.433 INFO:teuthology.orchestra.run.smithi032.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-24T17:23:38.433 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-24T17:23:38.433 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-24T17:23:38.433 INFO:teuthology.orchestra.run.smithi032.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-24T17:23:38.433 INFO:teuthology.orchestra.run.smithi032.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-24T17:23:38.433 INFO:teuthology.orchestra.run.smithi032.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-24T17:23:38.433 INFO:teuthology.orchestra.run.smithi032.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-24T17:23:38.433 INFO:teuthology.orchestra.run.smithi032.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-24T17:23:38.433 INFO:teuthology.orchestra.run.smithi032.stdout: xmlstarlet zip 2024-04-24T17:23:38.434 INFO:teuthology.orchestra.run.smithi032.stdout:Suggested packages: 2024-04-24T17:23:38.434 INFO:teuthology.orchestra.run.smithi032.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-24T17:23:38.434 INFO:teuthology.orchestra.run.smithi032.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-24T17:23:38.434 INFO:teuthology.orchestra.run.smithi032.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-24T17:23:38.434 INFO:teuthology.orchestra.run.smithi032.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-24T17:23:38.434 INFO:teuthology.orchestra.run.smithi032.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-24T17:23:38.434 INFO:teuthology.orchestra.run.smithi032.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-24T17:23:38.434 INFO:teuthology.orchestra.run.smithi032.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-24T17:23:38.434 INFO:teuthology.orchestra.run.smithi032.stdout:Recommended packages: 2024-04-24T17:23:38.434 INFO:teuthology.orchestra.run.smithi032.stdout: btrfs-tools 2024-04-24T17:23:38.510 INFO:teuthology.orchestra.run.smithi082.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-24T17:23:38.510 INFO:teuthology.orchestra.run.smithi082.stdout:Need to get 181 MB of archives. 2024-04-24T17:23:38.510 INFO:teuthology.orchestra.run.smithi082.stdout:After this operation, 829 MB of additional disk space will be used. 2024-04-24T17:23:38.510 INFO:teuthology.orchestra.run.smithi082.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-24T17:23:38.539 INFO:teuthology.orchestra.run.smithi032.stdout:The following NEW packages will be installed: 2024-04-24T17:23:38.539 INFO:teuthology.orchestra.run.smithi032.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-24T17:23:38.539 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-24T17:23:38.539 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-24T17:23:38.539 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-24T17:23:38.539 INFO:teuthology.orchestra.run.smithi032.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-24T17:23:38.540 INFO:teuthology.orchestra.run.smithi032.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-24T17:23:38.541 INFO:teuthology.orchestra.run.smithi032.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-24T17:23:38.541 INFO:teuthology.orchestra.run.smithi032.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-24T17:23:38.541 INFO:teuthology.orchestra.run.smithi032.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-24T17:23:38.541 INFO:teuthology.orchestra.run.smithi032.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-24T17:23:38.541 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-24T17:23:38.541 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-24T17:23:38.541 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-24T17:23:38.541 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-24T17:23:38.541 INFO:teuthology.orchestra.run.smithi032.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-24T17:23:38.541 INFO:teuthology.orchestra.run.smithi032.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-24T17:23:38.541 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-24T17:23:38.541 INFO:teuthology.orchestra.run.smithi032.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-24T17:23:38.541 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-24T17:23:38.541 INFO:teuthology.orchestra.run.smithi032.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-24T17:23:38.541 INFO:teuthology.orchestra.run.smithi032.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-24T17:23:38.542 INFO:teuthology.orchestra.run.smithi032.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-24T17:23:38.542 INFO:teuthology.orchestra.run.smithi032.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-24T17:23:38.542 INFO:teuthology.orchestra.run.smithi032.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-24T17:23:38.542 INFO:teuthology.orchestra.run.smithi032.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-24T17:23:38.542 INFO:teuthology.orchestra.run.smithi032.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-24T17:23:38.547 INFO:teuthology.orchestra.run.smithi082.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-3285-g5904d294-1jammy [3,556 kB] 2024-04-24T17:23:38.683 INFO:teuthology.orchestra.run.smithi032.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-24T17:23:38.684 INFO:teuthology.orchestra.run.smithi032.stdout:Need to get 181 MB of archives. 2024-04-24T17:23:38.684 INFO:teuthology.orchestra.run.smithi032.stdout:After this operation, 829 MB of additional disk space will be used. 2024-04-24T17:23:38.684 INFO:teuthology.orchestra.run.smithi032.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-3285-g5904d294-1jammy [3,556 kB] 2024-04-24T17:23:38.731 INFO:teuthology.orchestra.run.smithi032.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-24T17:23:38.940 INFO:teuthology.orchestra.run.smithi082.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-24T17:23:38.953 INFO:teuthology.orchestra.run.smithi082.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-24T17:23:39.036 INFO:teuthology.orchestra.run.smithi082.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-24T17:23:39.149 INFO:teuthology.orchestra.run.smithi032.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-3285-g5904d294-1jammy [3,224 kB] 2024-04-24T17:23:39.150 INFO:teuthology.orchestra.run.smithi032.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-24T17:23:39.166 INFO:teuthology.orchestra.run.smithi032.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-24T17:23:39.300 INFO:teuthology.orchestra.run.smithi032.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-24T17:23:39.325 INFO:teuthology.orchestra.run.smithi082.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-24T17:23:39.330 INFO:teuthology.orchestra.run.smithi082.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-24T17:23:39.341 INFO:teuthology.orchestra.run.smithi032.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-3285-g5904d294-1jammy [712 kB] 2024-04-24T17:23:39.346 INFO:teuthology.orchestra.run.smithi082.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-24T17:23:39.352 INFO:teuthology.orchestra.run.smithi082.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-24T17:23:39.353 INFO:teuthology.orchestra.run.smithi082.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-24T17:23:39.353 INFO:teuthology.orchestra.run.smithi082.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-24T17:23:39.379 INFO:teuthology.orchestra.run.smithi032.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-3285-g5904d294-1jammy [356 kB] 2024-04-24T17:23:39.389 INFO:teuthology.orchestra.run.smithi082.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-24T17:23:39.397 INFO:teuthology.orchestra.run.smithi032.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-3285-g5904d294-1jammy [32.8 kB] 2024-04-24T17:23:39.399 INFO:teuthology.orchestra.run.smithi032.stdout:Get:10 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-3285-g5904d294-1jammy [180 kB] 2024-04-24T17:23:39.403 INFO:teuthology.orchestra.run.smithi032.stdout:Get:11 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-3285-g5904d294-1jammy [67.4 kB] 2024-04-24T17:23:39.405 INFO:teuthology.orchestra.run.smithi032.stdout:Get:12 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-3285-g5904d294-1jammy [330 kB] 2024-04-24T17:23:39.421 INFO:teuthology.orchestra.run.smithi032.stdout:Get:13 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-3285-g5904d294-1jammy [6,893 kB] 2024-04-24T17:23:39.557 INFO:teuthology.orchestra.run.smithi082.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-24T17:23:39.601 INFO:teuthology.orchestra.run.smithi032.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-24T17:23:39.606 INFO:teuthology.orchestra.run.smithi032.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-24T17:23:39.624 INFO:teuthology.orchestra.run.smithi032.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-24T17:23:39.630 INFO:teuthology.orchestra.run.smithi032.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-24T17:23:39.631 INFO:teuthology.orchestra.run.smithi032.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-24T17:23:39.632 INFO:teuthology.orchestra.run.smithi032.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-24T17:23:39.654 INFO:teuthology.orchestra.run.smithi032.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-24T17:23:39.658 INFO:teuthology.orchestra.run.smithi032.stdout:Get:21 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-3285-g5904d294-1jammy [112 kB] 2024-04-24T17:23:39.661 INFO:teuthology.orchestra.run.smithi032.stdout:Get:22 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-3285-g5904d294-1jammy [466 kB] 2024-04-24T17:23:39.676 INFO:teuthology.orchestra.run.smithi032.stdout:Get:23 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-3285-g5904d294-1jammy [25.8 MB] 2024-04-24T17:23:39.831 INFO:teuthology.orchestra.run.smithi032.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-24T17:23:39.993 INFO:teuthology.orchestra.run.smithi082.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-24T17:23:40.031 INFO:teuthology.orchestra.run.smithi082.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-24T17:23:40.034 INFO:teuthology.orchestra.run.smithi082.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-24T17:23:40.135 INFO:teuthology.orchestra.run.smithi082.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-24T17:23:40.185 INFO:teuthology.orchestra.run.smithi082.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-24T17:23:40.215 INFO:teuthology.orchestra.run.smithi082.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-24T17:23:40.224 INFO:teuthology.orchestra.run.smithi082.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-24T17:23:40.226 INFO:teuthology.orchestra.run.smithi082.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-24T17:23:40.235 INFO:teuthology.orchestra.run.smithi032.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-24T17:23:40.239 INFO:teuthology.orchestra.run.smithi082.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-24T17:23:40.242 INFO:teuthology.orchestra.run.smithi082.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-24T17:23:40.274 INFO:teuthology.orchestra.run.smithi032.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-24T17:23:40.277 INFO:teuthology.orchestra.run.smithi032.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-24T17:23:40.351 INFO:teuthology.orchestra.run.smithi032.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-24T17:23:40.406 INFO:teuthology.orchestra.run.smithi032.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-24T17:23:40.408 INFO:teuthology.orchestra.run.smithi082.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-24T17:23:40.427 INFO:teuthology.orchestra.run.smithi032.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-24T17:23:40.433 INFO:teuthology.orchestra.run.smithi032.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-24T17:23:40.434 INFO:teuthology.orchestra.run.smithi032.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-24T17:23:40.445 INFO:teuthology.orchestra.run.smithi032.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-24T17:23:40.446 INFO:teuthology.orchestra.run.smithi032.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-24T17:23:40.491 INFO:teuthology.orchestra.run.smithi082.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-24T17:23:40.521 INFO:teuthology.orchestra.run.smithi082.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-24T17:23:40.538 INFO:teuthology.orchestra.run.smithi082.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-24T17:23:40.582 INFO:teuthology.orchestra.run.smithi082.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-24T17:23:40.614 INFO:teuthology.orchestra.run.smithi032.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-24T17:23:40.632 INFO:teuthology.orchestra.run.smithi032.stdout:Get:36 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-3285-g5904d294-1jammy [5,093 kB] 2024-04-24T17:23:40.712 INFO:teuthology.orchestra.run.smithi032.stdout:Get:37 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-3285-g5904d294-1jammy [244 kB] 2024-04-24T17:23:40.714 INFO:teuthology.orchestra.run.smithi032.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-24T17:23:40.715 INFO:teuthology.orchestra.run.smithi032.stdout:Get:39 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-3285-g5904d294-1jammy [124 kB] 2024-04-24T17:23:40.717 INFO:teuthology.orchestra.run.smithi032.stdout:Get:40 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-3285-g5904d294-1jammy [1,544 kB] 2024-04-24T17:23:40.745 INFO:teuthology.orchestra.run.smithi032.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-24T17:23:40.760 INFO:teuthology.orchestra.run.smithi032.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-24T17:23:40.768 INFO:teuthology.orchestra.run.smithi032.stdout:Get:43 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-3285-g5904d294-1jammy [6,257 kB] 2024-04-24T17:23:40.793 INFO:teuthology.orchestra.run.smithi082.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-24T17:23:40.804 INFO:teuthology.orchestra.run.smithi032.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-24T17:23:40.809 INFO:teuthology.orchestra.run.smithi082.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-24T17:23:40.816 INFO:teuthology.orchestra.run.smithi082.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-24T17:23:40.821 INFO:teuthology.orchestra.run.smithi082.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-24T17:23:40.843 INFO:teuthology.orchestra.run.smithi082.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-24T17:23:40.847 INFO:teuthology.orchestra.run.smithi082.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-24T17:23:40.988 INFO:teuthology.orchestra.run.smithi032.stdout:Get:45 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-3285-g5904d294-1jammy [22.8 MB] 2024-04-24T17:23:41.019 INFO:teuthology.orchestra.run.smithi082.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-24T17:23:41.020 INFO:teuthology.orchestra.run.smithi032.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-24T17:23:41.034 INFO:teuthology.orchestra.run.smithi032.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-24T17:23:41.041 INFO:teuthology.orchestra.run.smithi032.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-24T17:23:41.047 INFO:teuthology.orchestra.run.smithi032.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-24T17:23:41.069 INFO:teuthology.orchestra.run.smithi032.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-24T17:23:41.073 INFO:teuthology.orchestra.run.smithi032.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-24T17:23:41.251 INFO:teuthology.orchestra.run.smithi032.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-24T17:23:41.367 INFO:teuthology.orchestra.run.smithi082.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-24T17:23:41.400 INFO:teuthology.orchestra.run.smithi082.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-24T17:23:41.411 INFO:teuthology.orchestra.run.smithi082.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-24T17:23:41.629 INFO:teuthology.orchestra.run.smithi032.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-24T17:23:41.647 INFO:teuthology.orchestra.run.smithi082.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-24T17:23:41.647 INFO:teuthology.orchestra.run.smithi082.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-24T17:23:41.654 INFO:teuthology.orchestra.run.smithi082.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-24T17:23:41.662 INFO:teuthology.orchestra.run.smithi082.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-24T17:23:41.679 INFO:teuthology.orchestra.run.smithi082.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-24T17:23:41.683 INFO:teuthology.orchestra.run.smithi032.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-24T17:23:41.698 INFO:teuthology.orchestra.run.smithi032.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-24T17:23:41.703 INFO:teuthology.orchestra.run.smithi082.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-24T17:23:41.710 INFO:teuthology.orchestra.run.smithi082.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-24T17:23:41.804 INFO:teuthology.orchestra.run.smithi032.stdout:Get:56 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-3285-g5904d294-1jammy [14.1 kB] 2024-04-24T17:23:41.804 INFO:teuthology.orchestra.run.smithi032.stdout:Get:57 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-3285-g5904d294-1jammy [870 kB] 2024-04-24T17:23:41.822 INFO:teuthology.orchestra.run.smithi032.stdout:Get:58 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-3285-g5904d294-1jammy [2,437 kB] 2024-04-24T17:23:41.864 INFO:teuthology.orchestra.run.smithi032.stdout:Get:59 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-3285-g5904d294-1jammy [792 kB] 2024-04-24T17:23:41.874 INFO:teuthology.orchestra.run.smithi032.stdout:Get:60 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-3285-g5904d294-1jammy [149 kB] 2024-04-24T17:23:41.875 INFO:teuthology.orchestra.run.smithi032.stdout:Get:61 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-3285-g5904d294-1jammy [2,297 kB] 2024-04-24T17:23:41.892 INFO:teuthology.orchestra.run.smithi082.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-24T17:23:41.950 INFO:teuthology.orchestra.run.smithi032.stdout:Get:62 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-3285-g5904d294-1jammy [8,625 kB] 2024-04-24T17:23:41.979 INFO:teuthology.orchestra.run.smithi032.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-24T17:23:41.980 INFO:teuthology.orchestra.run.smithi032.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-24T17:23:41.986 INFO:teuthology.orchestra.run.smithi032.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-24T17:23:41.991 INFO:teuthology.orchestra.run.smithi032.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-24T17:23:42.011 INFO:teuthology.orchestra.run.smithi032.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-24T17:23:42.042 INFO:teuthology.orchestra.run.smithi032.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-24T17:23:42.051 INFO:teuthology.orchestra.run.smithi032.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-24T17:23:42.194 INFO:teuthology.orchestra.run.smithi082.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-24T17:23:42.241 INFO:teuthology.orchestra.run.smithi032.stdout:Get:70 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-3285-g5904d294-1jammy [14.2 kB] 2024-04-24T17:23:42.241 INFO:teuthology.orchestra.run.smithi032.stdout:Get:71 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-3285-g5904d294-1jammy [51.7 MB] 2024-04-24T17:23:42.247 INFO:teuthology.orchestra.run.smithi032.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-24T17:23:42.417 INFO:teuthology.orchestra.run.smithi082.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-24T17:23:42.437 INFO:teuthology.orchestra.run.smithi082.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-24T17:23:42.463 INFO:teuthology.orchestra.run.smithi082.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-24T17:23:42.542 INFO:teuthology.orchestra.run.smithi082.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-24T17:23:42.564 INFO:teuthology.orchestra.run.smithi032.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-24T17:23:42.794 INFO:teuthology.orchestra.run.smithi032.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-24T17:23:42.827 INFO:teuthology.orchestra.run.smithi032.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-24T17:23:42.857 INFO:teuthology.orchestra.run.smithi032.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-24T17:23:42.949 INFO:teuthology.orchestra.run.smithi032.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-24T17:23:42.959 INFO:teuthology.orchestra.run.smithi082.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-24T17:23:42.961 INFO:teuthology.orchestra.run.smithi082.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-24T17:23:42.965 INFO:teuthology.orchestra.run.smithi082.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-24T17:23:42.965 INFO:teuthology.orchestra.run.smithi082.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-24T17:23:42.968 INFO:teuthology.orchestra.run.smithi082.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-24T17:23:43.214 INFO:teuthology.orchestra.run.smithi082.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-24T17:23:43.571 INFO:teuthology.orchestra.run.smithi032.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-24T17:23:43.573 INFO:teuthology.orchestra.run.smithi032.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-24T17:23:43.576 INFO:teuthology.orchestra.run.smithi032.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-24T17:23:43.577 INFO:teuthology.orchestra.run.smithi032.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-24T17:23:43.580 INFO:teuthology.orchestra.run.smithi032.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-24T17:23:43.746 INFO:teuthology.orchestra.run.smithi082.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-24T17:23:43.752 INFO:teuthology.orchestra.run.smithi082.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-24T17:23:43.828 INFO:teuthology.orchestra.run.smithi032.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-24T17:23:44.184 INFO:teuthology.orchestra.run.smithi032.stdout:Get:84 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-3285-g5904d294-1jammy [124 kB] 2024-04-24T17:23:44.187 INFO:teuthology.orchestra.run.smithi032.stdout:Get:85 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-3285-g5904d294-1jammy [38.9 kB] 2024-04-24T17:23:44.188 INFO:teuthology.orchestra.run.smithi032.stdout:Get:86 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-3285-g5904d294-1jammy [13.6 MB] 2024-04-24T17:23:44.269 INFO:teuthology.orchestra.run.smithi032.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-24T17:23:44.275 INFO:teuthology.orchestra.run.smithi032.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-24T17:23:44.291 INFO:teuthology.orchestra.run.smithi082.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-24T17:23:44.291 INFO:teuthology.orchestra.run.smithi082.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-24T17:23:44.293 INFO:teuthology.orchestra.run.smithi082.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-24T17:23:44.294 INFO:teuthology.orchestra.run.smithi082.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-24T17:23:44.297 INFO:teuthology.orchestra.run.smithi082.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-24T17:23:44.297 INFO:teuthology.orchestra.run.smithi082.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-24T17:23:44.298 INFO:teuthology.orchestra.run.smithi082.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-24T17:23:44.299 INFO:teuthology.orchestra.run.smithi082.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-24T17:23:44.488 INFO:teuthology.orchestra.run.smithi082.stdout:Get:68 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-3285-g5904d294-1jammy [3,224 kB] 2024-04-24T17:23:44.508 INFO:teuthology.orchestra.run.smithi082.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-24T17:23:44.582 INFO:teuthology.orchestra.run.smithi032.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-24T17:23:44.582 INFO:teuthology.orchestra.run.smithi032.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-3285-g5904d294-1jammy [91.4 kB] 2024-04-24T17:23:44.583 INFO:teuthology.orchestra.run.smithi032.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-24T17:23:44.619 INFO:teuthology.orchestra.run.smithi032.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-24T17:23:44.620 INFO:teuthology.orchestra.run.smithi032.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-24T17:23:44.621 INFO:teuthology.orchestra.run.smithi032.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-24T17:23:44.622 INFO:teuthology.orchestra.run.smithi032.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-24T17:23:44.623 INFO:teuthology.orchestra.run.smithi032.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-24T17:23:44.624 INFO:teuthology.orchestra.run.smithi032.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-24T17:23:44.799 INFO:teuthology.orchestra.run.smithi032.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-24T17:23:44.922 INFO:teuthology.orchestra.run.smithi082.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-24T17:23:45.013 INFO:teuthology.orchestra.run.smithi082.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-24T17:23:45.034 INFO:teuthology.orchestra.run.smithi082.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-24T17:23:45.140 INFO:teuthology.orchestra.run.smithi082.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-24T17:23:45.185 INFO:teuthology.orchestra.run.smithi082.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-24T17:23:45.196 INFO:teuthology.orchestra.run.smithi082.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-24T17:23:45.202 INFO:teuthology.orchestra.run.smithi082.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-24T17:23:45.225 INFO:teuthology.orchestra.run.smithi032.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-24T17:23:45.255 INFO:teuthology.orchestra.run.smithi082.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-24T17:23:45.259 INFO:teuthology.orchestra.run.smithi082.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-24T17:23:45.279 INFO:teuthology.orchestra.run.smithi032.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-24T17:23:45.280 INFO:teuthology.orchestra.run.smithi082.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-24T17:23:45.306 INFO:teuthology.orchestra.run.smithi032.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-24T17:23:45.460 INFO:teuthology.orchestra.run.smithi032.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-24T17:23:45.515 INFO:teuthology.orchestra.run.smithi082.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-24T17:23:45.532 INFO:teuthology.orchestra.run.smithi032.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-24T17:23:45.548 INFO:teuthology.orchestra.run.smithi032.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-24T17:23:45.555 INFO:teuthology.orchestra.run.smithi032.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-24T17:23:45.647 INFO:teuthology.orchestra.run.smithi032.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-24T17:23:45.650 INFO:teuthology.orchestra.run.smithi032.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-24T17:23:45.687 INFO:teuthology.orchestra.run.smithi032.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-24T17:23:45.852 INFO:teuthology.orchestra.run.smithi032.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-24T17:23:45.923 INFO:teuthology.orchestra.run.smithi082.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-24T17:23:45.954 INFO:teuthology.orchestra.run.smithi082.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-24T17:23:46.116 INFO:teuthology.orchestra.run.smithi032.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-24T17:23:46.134 INFO:teuthology.orchestra.run.smithi032.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-24T17:23:46.192 INFO:teuthology.orchestra.run.smithi082.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-24T17:23:46.244 INFO:teuthology.orchestra.run.smithi082.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-24T17:23:46.286 INFO:teuthology.orchestra.run.smithi032.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-24T17:23:46.315 INFO:teuthology.orchestra.run.smithi032.stdout:Get:113 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-24T17:23:46.473 INFO:teuthology.orchestra.run.smithi032.stdout:Get:114 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-24T17:23:46.475 INFO:teuthology.orchestra.run.smithi032.stdout:Get:115 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-24T17:23:46.477 INFO:teuthology.orchestra.run.smithi032.stdout:Get:116 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-24T17:23:46.499 INFO:teuthology.orchestra.run.smithi032.stdout:Get:117 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-24T17:23:46.504 INFO:teuthology.orchestra.run.smithi032.stdout:Get:118 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-24T17:23:46.640 INFO:teuthology.orchestra.run.smithi082.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-24T17:23:46.647 INFO:teuthology.orchestra.run.smithi082.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-24T17:23:46.656 INFO:teuthology.orchestra.run.smithi082.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-24T17:23:46.724 INFO:teuthology.orchestra.run.smithi082.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-24T17:23:46.744 INFO:teuthology.orchestra.run.smithi082.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-24T17:23:46.917 INFO:teuthology.orchestra.run.smithi082.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-3285-g5904d294-1jammy [712 kB] 2024-04-24T17:23:47.194 INFO:teuthology.orchestra.run.smithi032.stdout:Fetched 181 MB in 8s (22.5 MB/s) 2024-04-24T17:23:47.308 INFO:teuthology.orchestra.run.smithi082.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-3285-g5904d294-1jammy [356 kB] 2024-04-24T17:23:47.328 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-24T17:23:47.430 INFO:teuthology.orchestra.run.smithi032.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-24T17:23:47.435 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-24T17:23:47.481 INFO:teuthology.orchestra.run.smithi082.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-3285-g5904d294-1jammy [32.8 kB] 2024-04-24T17:23:47.484 INFO:teuthology.orchestra.run.smithi082.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-3285-g5904d294-1jammy [180 kB] 2024-04-24T17:23:47.518 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-24T17:23:47.558 INFO:teuthology.orchestra.run.smithi082.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-3285-g5904d294-1jammy [67.4 kB] 2024-04-24T17:23:47.592 INFO:teuthology.orchestra.run.smithi082.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-3285-g5904d294-1jammy [330 kB] 2024-04-24T17:23:47.747 INFO:teuthology.orchestra.run.smithi082.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-3285-g5904d294-1jammy [6,893 kB] 2024-04-24T17:23:47.880 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-24T17:23:47.907 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-24T17:23:47.946 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-24T17:23:48.324 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-24T17:23:48.351 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-24T17:23:48.382 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-24T17:23:48.794 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-24T17:23:48.822 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-24T17:23:48.870 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T17:23:49.247 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-24T17:23:49.275 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-24T17:23:49.314 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T17:23:49.733 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-24T17:23:49.760 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-24T17:23:49.791 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T17:23:50.111 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-24T17:23:50.139 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-24T17:23:50.178 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-24T17:23:50.210 INFO:teuthology.orchestra.run.smithi082.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-3285-g5904d294-1jammy [112 kB] 2024-04-24T17:23:50.238 INFO:teuthology.orchestra.run.smithi082.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-3285-g5904d294-1jammy [466 kB] 2024-04-24T17:23:50.316 INFO:teuthology.orchestra.run.smithi082.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-3285-g5904d294-1jammy [25.8 MB] 2024-04-24T17:23:50.489 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package librados2. 2024-04-24T17:23:50.516 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../007-librados2_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:23:50.555 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking librados2 (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:23:50.983 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package librbd1. 2024-04-24T17:23:51.010 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../008-librbd1_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:23:51.049 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking librbd1 (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:23:51.494 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libcephfs2. 2024-04-24T17:23:51.521 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:23:51.561 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libcephfs2 (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:23:52.022 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-rados. 2024-04-24T17:23:52.048 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../010-python3-rados_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:23:52.080 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-rados (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:23:52.374 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-24T17:23:52.400 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-3285-g5904d294-1jammy_all.deb ... 2024-04-24T17:23:52.440 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-ceph-argparse (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:23:52.693 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-cephfs. 2024-04-24T17:23:52.719 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:23:52.759 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-cephfs (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:23:53.079 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-24T17:23:53.105 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-3285-g5904d294-1jammy_all.deb ... 2024-04-24T17:23:53.116 INFO:teuthology.orchestra.run.smithi082.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-3285-g5904d294-1jammy [5,093 kB] 2024-04-24T17:23:53.145 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-ceph-common (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:23:53.390 INFO:teuthology.orchestra.run.smithi082.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-3285-g5904d294-1jammy [244 kB] 2024-04-24T17:23:53.418 INFO:teuthology.orchestra.run.smithi082.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-3285-g5904d294-1jammy [124 kB] 2024-04-24T17:23:53.421 INFO:teuthology.orchestra.run.smithi082.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-3285-g5904d294-1jammy [1,544 kB] 2024-04-24T17:23:53.465 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-24T17:23:53.492 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-24T17:23:53.523 INFO:teuthology.orchestra.run.smithi082.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-3285-g5904d294-1jammy [6,257 kB] 2024-04-24T17:23:53.531 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-24T17:23:53.792 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-prettytable. 2024-04-24T17:23:53.808 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-24T17:23:53.842 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-24T17:23:53.860 INFO:teuthology.orchestra.run.smithi082.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-3285-g5904d294-1jammy [22.8 MB] 2024-04-24T17:23:54.112 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-rbd. 2024-04-24T17:23:54.128 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:23:54.170 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-rbd (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:23:54.514 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-24T17:23:54.541 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-24T17:23:54.581 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-24T17:23:54.858 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-24T17:23:54.874 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-24T17:23:54.916 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-24T17:23:55.036 INFO:teuthology.orchestra.run.smithi082.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-3285-g5904d294-1jammy [14.1 kB] 2024-04-24T17:23:55.037 INFO:teuthology.orchestra.run.smithi082.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-3285-g5904d294-1jammy [870 kB] 2024-04-24T17:23:55.053 INFO:teuthology.orchestra.run.smithi082.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-3285-g5904d294-1jammy [2,437 kB] 2024-04-24T17:23:55.120 INFO:teuthology.orchestra.run.smithi082.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-3285-g5904d294-1jammy [792 kB] 2024-04-24T17:23:55.158 INFO:teuthology.orchestra.run.smithi082.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-3285-g5904d294-1jammy [149 kB] 2024-04-24T17:23:55.161 INFO:teuthology.orchestra.run.smithi082.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-3285-g5904d294-1jammy [2,297 kB] 2024-04-24T17:23:55.171 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-24T17:23:55.187 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-24T17:23:55.230 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-24T17:23:55.253 INFO:teuthology.orchestra.run.smithi082.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-3285-g5904d294-1jammy [8,625 kB] 2024-04-24T17:23:55.562 INFO:teuthology.orchestra.run.smithi082.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-3285-g5904d294-1jammy [14.2 kB] 2024-04-24T17:23:55.563 INFO:teuthology.orchestra.run.smithi082.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-3285-g5904d294-1jammy [51.7 MB] 2024-04-24T17:23:55.566 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package lua5.1. 2024-04-24T17:23:55.593 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-24T17:23:55.632 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-24T17:23:55.902 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package lua-any. 2024-04-24T17:23:55.918 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-24T17:23:55.952 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-24T17:23:56.129 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package zip. 2024-04-24T17:23:56.145 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-24T17:23:56.196 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking zip (3.0-12build2) ... 2024-04-24T17:23:56.541 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package unzip. 2024-04-24T17:23:56.568 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-24T17:23:56.607 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-24T17:23:57.003 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package luarocks. 2024-04-24T17:23:57.029 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-24T17:23:57.069 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-24T17:23:57.452 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package librgw2. 2024-04-24T17:23:57.478 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../025-librgw2_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:23:57.518 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking librgw2 (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:23:57.701 INFO:teuthology.orchestra.run.smithi082.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-3285-g5904d294-1jammy [124 kB] 2024-04-24T17:23:57.733 INFO:teuthology.orchestra.run.smithi082.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-3285-g5904d294-1jammy [38.9 kB] 2024-04-24T17:23:57.735 INFO:teuthology.orchestra.run.smithi082.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-3285-g5904d294-1jammy [13.6 MB] 2024-04-24T17:23:58.045 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-rgw. 2024-04-24T17:23:58.061 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:23:58.091 INFO:teuthology.orchestra.run.smithi082.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-3285-g5904d294-1jammy [91.4 kB] 2024-04-24T17:23:58.095 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-rgw (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:23:58.432 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-24T17:23:58.458 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-24T17:23:58.498 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-24T17:23:58.718 INFO:teuthology.orchestra.run.smithi082.stdout:Fetched 181 MB in 20s (9,134 kB/s) 2024-04-24T17:23:58.809 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libradosstriper1. 2024-04-24T17:23:58.836 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:23:58.845 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-24T17:23:58.876 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libradosstriper1 (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:23:58.943 INFO:teuthology.orchestra.run.smithi082.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-24T17:23:58.948 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-24T17:23:59.028 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-24T17:23:59.237 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-common. 2024-04-24T17:23:59.264 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../029-ceph-common_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:23:59.304 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-common (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:23:59.322 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-24T17:23:59.348 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-24T17:23:59.389 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-24T17:23:59.725 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-24T17:23:59.752 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-24T17:23:59.792 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-24T17:24:00.229 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-24T17:24:00.255 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-24T17:24:00.303 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T17:24:00.707 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-24T17:24:00.725 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-base. 2024-04-24T17:24:00.733 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-24T17:24:00.752 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../030-ceph-base_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:24:00.773 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T17:24:00.801 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-base (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:01.110 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-24T17:24:01.136 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-24T17:24:01.176 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T17:24:01.363 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-24T17:24:01.389 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-24T17:24:01.429 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-24T17:24:01.546 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-24T17:24:01.572 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-24T17:24:01.612 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-24T17:24:01.732 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-cheroot. 2024-04-24T17:24:01.759 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-24T17:24:01.798 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-24T17:24:01.882 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package librados2. 2024-04-24T17:24:01.897 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../007-librados2_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:24:01.940 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking librados2 (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:02.093 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-24T17:24:02.120 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-24T17:24:02.151 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-24T17:24:02.393 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package librbd1. 2024-04-24T17:24:02.419 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../008-librbd1_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:24:02.459 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking librbd1 (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:02.579 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-24T17:24:02.606 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-24T17:24:02.645 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-24T17:24:02.896 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libcephfs2. 2024-04-24T17:24:02.922 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:24:02.932 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-24T17:24:02.958 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-24T17:24:02.962 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libcephfs2 (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:02.998 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-24T17:24:03.231 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-rados. 2024-04-24T17:24:03.246 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../010-python3-rados_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:24:03.281 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-rados (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:03.301 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-tempora. 2024-04-24T17:24:03.328 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-24T17:24:03.367 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-24T17:24:03.629 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-portend. 2024-04-24T17:24:03.655 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-24T17:24:03.687 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-24T17:24:03.746 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-24T17:24:03.772 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-3285-g5904d294-1jammy_all.deb ... 2024-04-24T17:24:03.817 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-ceph-argparse (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:03.973 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-24T17:24:04.000 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-24T17:24:04.040 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-24T17:24:04.070 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-cephfs. 2024-04-24T17:24:04.085 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:24:04.128 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-cephfs (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:04.342 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-24T17:24:04.369 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-24T17:24:04.409 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-24T17:24:04.439 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-24T17:24:04.465 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-3285-g5904d294-1jammy_all.deb ... 2024-04-24T17:24:04.506 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-ceph-common (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:04.720 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-natsort. 2024-04-24T17:24:04.747 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-24T17:24:04.779 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-24T17:24:04.842 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-24T17:24:04.868 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-24T17:24:04.909 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-24T17:24:05.090 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-logutils. 2024-04-24T17:24:05.117 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-24T17:24:05.157 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-24T17:24:05.211 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-prettytable. 2024-04-24T17:24:05.238 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-24T17:24:05.270 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-24T17:24:05.459 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-24T17:24:05.486 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-24T17:24:05.522 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-rbd. 2024-04-24T17:24:05.526 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-24T17:24:05.548 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:24:05.589 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-rbd (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:05.829 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-mako. 2024-04-24T17:24:05.855 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-24T17:24:05.895 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-24T17:24:05.942 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-24T17:24:05.968 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-24T17:24:06.008 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-24T17:24:06.198 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-24T17:24:06.214 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-24T17:24:06.248 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-24T17:24:06.294 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-24T17:24:06.309 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-24T17:24:06.353 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-24T17:24:06.484 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-24T17:24:06.511 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-24T17:24:06.552 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-24T17:24:06.656 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-24T17:24:06.682 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-24T17:24:06.714 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-24T17:24:06.862 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-webob. 2024-04-24T17:24:06.889 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-24T17:24:06.928 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-24T17:24:07.026 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package lua5.1. 2024-04-24T17:24:07.052 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-24T17:24:07.092 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-24T17:24:07.256 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-waitress. 2024-04-24T17:24:07.283 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-24T17:24:07.326 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-24T17:24:07.403 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package lua-any. 2024-04-24T17:24:07.430 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-24T17:24:07.470 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-24T17:24:07.642 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-tempita. 2024-04-24T17:24:07.669 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-24T17:24:07.706 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package zip. 2024-04-24T17:24:07.709 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-24T17:24:07.732 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-24T17:24:07.772 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking zip (3.0-12build2) ... 2024-04-24T17:24:07.978 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-paste. 2024-04-24T17:24:08.005 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-24T17:24:08.045 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-24T17:24:08.050 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package unzip. 2024-04-24T17:24:08.076 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-24T17:24:08.117 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-24T17:24:08.381 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-24T17:24:08.408 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-24T17:24:08.448 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-24T17:24:08.504 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package luarocks. 2024-04-24T17:24:08.530 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-24T17:24:08.570 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-24T17:24:08.726 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-24T17:24:08.753 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-24T17:24:08.792 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-24T17:24:09.061 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-webtest. 2024-04-24T17:24:09.077 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-24T17:24:09.111 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-24T17:24:09.129 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package librgw2. 2024-04-24T17:24:09.155 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../025-librgw2_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:24:09.195 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking librgw2 (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:09.431 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-pecan. 2024-04-24T17:24:09.458 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-24T17:24:09.498 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-24T17:24:09.722 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-rgw. 2024-04-24T17:24:09.749 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:24:09.781 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-rgw (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:09.852 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-certifi. 2024-04-24T17:24:09.879 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-24T17:24:09.918 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-24T17:24:10.100 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-24T17:24:10.127 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-24T17:24:10.167 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-24T17:24:10.263 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-idna. 2024-04-24T17:24:10.290 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-24T17:24:10.329 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-24T17:24:10.478 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libradosstriper1. 2024-04-24T17:24:10.504 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:24:10.544 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libradosstriper1 (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:10.649 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-urllib3. 2024-04-24T17:24:10.676 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-24T17:24:10.716 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-24T17:24:10.940 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-common. 2024-04-24T17:24:10.966 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../029-ceph-common_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:24:11.006 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-common (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:11.043 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-requests. 2024-04-24T17:24:11.070 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-24T17:24:11.110 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-24T17:24:11.388 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-24T17:24:11.415 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-24T17:24:11.446 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-24T17:24:12.534 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-24T17:24:12.561 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-3285-g5904d294-1jammy_all.deb ... 2024-04-24T17:24:12.599 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-mgr-modules-core (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:13.070 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-base. 2024-04-24T17:24:13.096 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../030-ceph-base_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:24:13.146 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-base (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:13.162 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-24T17:24:13.189 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:24:13.229 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:13.523 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-24T17:24:13.550 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-24T17:24:13.590 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-24T17:24:13.715 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-24T17:24:13.742 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-24T17:24:13.782 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-24T17:24:13.868 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-openssl. 2024-04-24T17:24:13.895 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-24T17:24:13.926 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-24T17:24:14.093 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-cheroot. 2024-04-24T17:24:14.120 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-24T17:24:14.160 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-24T17:24:14.262 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-mgr. 2024-04-24T17:24:14.289 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:24:14.328 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-mgr (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:14.496 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-24T17:24:14.522 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-24T17:24:14.562 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-24T17:24:14.723 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-mon. 2024-04-24T17:24:14.750 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:24:14.790 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-mon (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:14.815 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-24T17:24:14.842 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-24T17:24:14.882 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-24T17:24:15.185 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-24T17:24:15.211 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-24T17:24:15.252 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-24T17:24:15.326 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-osd. 2024-04-24T17:24:15.354 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:24:15.384 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-osd (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:15.538 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-tempora. 2024-04-24T17:24:15.564 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-24T17:24:15.605 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-24T17:24:15.873 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-portend. 2024-04-24T17:24:15.889 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-24T17:24:15.923 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-24T17:24:16.210 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-24T17:24:16.236 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-24T17:24:16.276 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-24T17:24:16.421 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph. 2024-04-24T17:24:16.448 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../066-ceph_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:24:16.487 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:16.579 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-24T17:24:16.606 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-24T17:24:16.662 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-24T17:24:16.824 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-fuse. 2024-04-24T17:24:16.851 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:24:16.890 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-fuse (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:16.983 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-natsort. 2024-04-24T17:24:17.009 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-24T17:24:17.049 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-24T17:24:17.251 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-mds. 2024-04-24T17:24:17.278 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:24:17.318 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-mds (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:17.361 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-logutils. 2024-04-24T17:24:17.387 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-24T17:24:17.427 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-24T17:24:17.687 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package cephadm. 2024-04-24T17:24:17.703 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../069-cephadm_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:24:17.705 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-24T17:24:17.731 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-24T17:24:17.737 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking cephadm (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:17.763 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-24T17:24:18.049 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-mako. 2024-04-24T17:24:18.057 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-24T17:24:18.075 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-24T17:24:18.084 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-24T17:24:18.123 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-24T17:24:18.124 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-24T17:24:18.400 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-24T17:24:18.416 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-24T17:24:18.450 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-24T17:24:18.461 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-24T17:24:18.487 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-24T17:24:18.527 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-24T17:24:18.830 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-24T17:24:18.856 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-24T17:24:18.897 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-24T17:24:19.166 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-webob. 2024-04-24T17:24:19.192 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-24T17:24:19.225 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-24T17:24:19.430 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-babel. 2024-04-24T17:24:19.458 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-24T17:24:19.496 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-24T17:24:19.527 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-waitress. 2024-04-24T17:24:19.554 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-24T17:24:19.588 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-24T17:24:19.825 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-jinja2. 2024-04-24T17:24:19.852 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-24T17:24:19.889 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-tempita. 2024-04-24T17:24:19.891 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-24T17:24:19.915 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-24T17:24:19.955 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-24T17:24:20.219 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-24T17:24:20.246 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-3285-g5904d294-1jammy_all.deb ... 2024-04-24T17:24:20.266 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-paste. 2024-04-24T17:24:20.285 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-mgr-cephadm (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:20.293 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-24T17:24:20.333 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-24T17:24:20.589 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-24T17:24:20.616 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-24T17:24:20.644 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-24T17:24:20.655 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-24T17:24:20.668 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-24T17:24:20.711 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-24T17:24:20.958 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-routes. 2024-04-24T17:24:20.986 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-24T17:24:21.013 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-24T17:24:21.024 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-24T17:24:21.040 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-24T17:24:21.080 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-24T17:24:21.361 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-24T17:24:21.383 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-webtest. 2024-04-24T17:24:21.388 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-3285-g5904d294-1jammy_all.deb ... 2024-04-24T17:24:21.410 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-24T17:24:21.427 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-mgr-dashboard (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:21.450 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-24T17:24:21.744 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-pecan. 2024-04-24T17:24:21.771 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-24T17:24:21.811 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-24T17:24:22.132 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-certifi. 2024-04-24T17:24:22.159 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-24T17:24:22.190 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-24T17:24:22.502 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-idna. 2024-04-24T17:24:22.528 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-24T17:24:22.568 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-24T17:24:22.691 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-24T17:24:22.718 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-24T17:24:22.758 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-24T17:24:22.879 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-urllib3. 2024-04-24T17:24:22.906 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-24T17:24:22.946 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-24T17:24:23.177 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-joblib. 2024-04-24T17:24:23.205 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-24T17:24:23.235 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-24T17:24:23.266 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-requests. 2024-04-24T17:24:23.292 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-24T17:24:23.332 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-24T17:24:23.613 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-24T17:24:23.641 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-24T17:24:23.652 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-24T17:24:23.678 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-24T17:24:23.679 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-24T17:24:23.718 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-24T17:24:23.958 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-sklearn. 2024-04-24T17:24:23.985 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-24T17:24:24.016 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-24T17:24:24.021 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-24T17:24:24.048 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-3285-g5904d294-1jammy_all.deb ... 2024-04-24T17:24:24.088 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-mgr-modules-core (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:24.476 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-24T17:24:24.503 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:24:24.580 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:24.724 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-24T17:24:24.752 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-3285-g5904d294-1jammy_all.deb ... 2024-04-24T17:24:24.876 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:25.012 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-24T17:24:25.039 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-24T17:24:25.078 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-24T17:24:25.398 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-openssl. 2024-04-24T17:24:25.424 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-24T17:24:25.465 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-24T17:24:25.743 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-mgr. 2024-04-24T17:24:25.769 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:24:25.801 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-mgr (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:25.846 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-cachetools. 2024-04-24T17:24:25.874 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-24T17:24:25.912 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-24T17:24:26.196 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-mon. 2024-04-24T17:24:26.215 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-24T17:24:26.222 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:24:26.243 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-24T17:24:26.262 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-mon (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:26.282 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-24T17:24:26.618 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-24T17:24:26.646 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-24T17:24:26.685 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-24T17:24:26.857 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-osd. 2024-04-24T17:24:26.884 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:24:26.923 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-osd (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:27.029 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-rsa. 2024-04-24T17:24:27.057 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-24T17:24:27.096 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-24T17:24:27.366 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-google-auth. 2024-04-24T17:24:27.394 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-24T17:24:27.457 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-24T17:24:27.752 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-24T17:24:27.780 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-24T17:24:27.810 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-24T17:24:27.952 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph. 2024-04-24T17:24:27.967 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../066-ceph_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:24:28.001 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:28.113 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-websocket. 2024-04-24T17:24:28.141 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-24T17:24:28.180 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-24T17:24:28.330 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-fuse. 2024-04-24T17:24:28.356 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:24:28.396 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-fuse (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:28.742 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-24T17:24:28.766 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-mds. 2024-04-24T17:24:28.769 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-24T17:24:28.793 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:24:28.808 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-24T17:24:28.832 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-mds (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:29.219 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package cephadm. 2024-04-24T17:24:29.246 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../069-cephadm_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:24:29.285 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking cephadm (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:29.388 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-24T17:24:29.406 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-3285-g5904d294-1jammy_all.deb ... 2024-04-24T17:24:29.446 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:29.538 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-24T17:24:29.565 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-24T17:24:29.604 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-24T17:24:29.774 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-24T17:24:29.802 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-24T17:24:29.840 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-24T17:24:29.898 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-24T17:24:29.914 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-24T17:24:29.948 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-24T17:24:30.102 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-24T17:24:30.130 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-24T17:24:30.168 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-24T17:24:30.429 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package jq. 2024-04-24T17:24:30.446 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-24T17:24:30.487 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-24T17:24:30.740 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package socat. 2024-04-24T17:24:30.768 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-24T17:24:30.806 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-24T17:24:30.945 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-babel. 2024-04-24T17:24:30.973 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-24T17:24:31.003 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-24T17:24:31.110 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package xmlstarlet. 2024-04-24T17:24:31.137 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-24T17:24:31.184 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-24T17:24:31.323 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-jinja2. 2024-04-24T17:24:31.350 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-24T17:24:31.389 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-24T17:24:31.538 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-test. 2024-04-24T17:24:31.555 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../097-ceph-test_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:24:31.596 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-test (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:31.734 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-24T17:24:31.761 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-3285-g5904d294-1jammy_all.deb ... 2024-04-24T17:24:31.801 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-mgr-cephadm (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:32.162 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-24T17:24:32.189 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-24T17:24:32.229 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-24T17:24:32.515 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-routes. 2024-04-24T17:24:32.542 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-24T17:24:32.581 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-24T17:24:32.901 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-24T17:24:32.928 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-3285-g5904d294-1jammy_all.deb ... 2024-04-24T17:24:32.968 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-mgr-dashboard (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:34.018 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package ceph-volume. 2024-04-24T17:24:34.045 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-3285-g5904d294-1jammy_all.deb ... 2024-04-24T17:24:34.084 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking ceph-volume (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:34.147 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-24T17:24:34.163 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-24T17:24:34.197 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-24T17:24:34.346 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-24T17:24:34.362 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:24:34.395 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking libcephfs-dev (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:34.643 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-joblib. 2024-04-24T17:24:34.658 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-24T17:24:34.692 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-24T17:24:34.698 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-24T17:24:34.726 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-24T17:24:34.765 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-24T17:24:35.096 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-24T17:24:35.124 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-24T17:24:35.163 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-24T17:24:35.271 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-24T17:24:35.298 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-24T17:24:35.329 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-24T17:24:35.450 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package nvme-cli. 2024-04-24T17:24:35.478 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-24T17:24:35.533 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-24T17:24:35.573 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-sklearn. 2024-04-24T17:24:35.600 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-24T17:24:35.639 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-24T17:24:35.913 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package pkg-config. 2024-04-24T17:24:35.941 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-24T17:24:35.980 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-24T17:24:36.191 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-24T17:24:36.207 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-24T17:24:36.241 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-24T17:24:36.392 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-24T17:24:36.420 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-3285-g5904d294-1jammy_all.deb ... 2024-04-24T17:24:36.535 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:36.619 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-24T17:24:36.648 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-24T17:24:36.677 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-24T17:24:36.963 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-pastescript. 2024-04-24T17:24:36.991 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-24T17:24:37.030 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-24T17:24:37.366 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-pluggy. 2024-04-24T17:24:37.394 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-24T17:24:37.433 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-24T17:24:37.463 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-cachetools. 2024-04-24T17:24:37.490 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-24T17:24:37.530 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-24T17:24:37.711 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-psutil. 2024-04-24T17:24:37.739 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-24T17:24:37.769 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-24T17:24:37.833 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-24T17:24:37.860 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-24T17:24:37.899 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-24T17:24:38.080 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-py. 2024-04-24T17:24:38.108 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-24T17:24:38.146 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-24T17:24:38.194 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-24T17:24:38.210 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-24T17:24:38.244 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-24T17:24:38.425 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-pygments. 2024-04-24T17:24:38.441 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-24T17:24:38.474 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-24T17:24:38.513 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-rsa. 2024-04-24T17:24:38.540 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-24T17:24:38.580 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-24T17:24:38.900 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-google-auth. 2024-04-24T17:24:38.927 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-24T17:24:38.966 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-24T17:24:39.154 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-24T17:24:39.182 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-24T17:24:39.220 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-24T17:24:39.286 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-24T17:24:39.313 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-24T17:24:39.353 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-24T17:24:39.523 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-toml. 2024-04-24T17:24:39.551 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-24T17:24:39.589 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-24T17:24:39.689 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-websocket. 2024-04-24T17:24:39.716 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-24T17:24:39.755 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-24T17:24:39.900 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-pytest. 2024-04-24T17:24:39.929 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-24T17:24:39.967 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-24T17:24:40.059 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-24T17:24:40.086 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-24T17:24:40.116 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-24T17:24:40.321 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package python3-simplejson. 2024-04-24T17:24:40.349 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-24T17:24:40.387 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-24T17:24:40.639 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-24T17:24:40.656 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-24T17:24:40.689 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-24T17:24:40.755 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-24T17:24:40.772 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-3285-g5904d294-1jammy_all.deb ... 2024-04-24T17:24:40.813 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:41.133 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-24T17:24:41.161 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-24T17:24:41.199 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-24T17:24:41.343 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package radosgw. 2024-04-24T17:24:41.372 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../116-radosgw_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:24:41.410 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking radosgw (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:41.503 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-24T17:24:41.529 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-24T17:24:41.569 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-24T17:24:41.822 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package jq. 2024-04-24T17:24:41.849 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-24T17:24:41.888 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-24T17:24:42.140 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package socat. 2024-04-24T17:24:42.156 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-24T17:24:42.198 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-24T17:24:42.321 INFO:teuthology.orchestra.run.smithi032.stdout:Selecting previously unselected package rbd-fuse. 2024-04-24T17:24:42.338 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:24:42.379 INFO:teuthology.orchestra.run.smithi032.stdout:Unpacking rbd-fuse (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:42.519 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package xmlstarlet. 2024-04-24T17:24:42.546 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-24T17:24:42.585 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-24T17:24:42.668 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-24T17:24:42.917 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-24T17:24:42.922 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-test. 2024-04-24T17:24:42.950 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../097-ceph-test_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:24:42.989 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-test (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:43.027 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-24T17:24:43.272 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-24T17:24:43.627 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-24T17:24:44.050 INFO:teuthology.orchestra.run.smithi032.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-24T17:24:44.112 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up cephadm (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:44.745 INFO:teuthology.orchestra.run.smithi032.stdout:Adding system user cephadm....done 2024-04-24T17:24:44.860 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-24T17:24:45.145 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-24T17:24:45.396 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-24T17:24:45.522 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-24T17:24:45.532 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-volume. 2024-04-24T17:24:45.559 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-3285-g5904d294-1jammy_all.deb ... 2024-04-24T17:24:45.601 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-volume (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:45.781 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-24T17:24:45.913 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-24T17:24:45.941 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:24:45.979 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libcephfs-dev (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:46.049 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-24T17:24:46.175 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-24T17:24:46.299 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-24T17:24:46.327 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-24T17:24:46.365 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-24T17:24:46.519 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-24T17:24:46.653 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-24T17:24:46.669 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-24T17:24:46.703 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-24T17:24:46.871 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-24T17:24:47.065 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package nvme-cli. 2024-04-24T17:24:47.092 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-24T17:24:47.131 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-24T17:24:47.138 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-24T17:24:47.414 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-24T17:24:47.561 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package pkg-config. 2024-04-24T17:24:47.589 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-24T17:24:47.628 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-24T17:24:47.658 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-24T17:24:47.889 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-24T17:24:47.905 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-24T17:24:47.917 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-24T17:24:47.939 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-24T17:24:48.177 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-24T17:24:48.342 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-24T17:24:48.369 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-24T17:24:48.408 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-24T17:24:48.419 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-ceph-argparse (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:48.669 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-pastescript. 2024-04-24T17:24:48.695 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-24T17:24:48.697 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-24T17:24:48.727 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-24T17:24:48.821 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-24T17:24:48.947 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-24T17:24:49.022 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-pluggy. 2024-04-24T17:24:49.050 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-24T17:24:49.073 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-24T17:24:49.088 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-24T17:24:49.341 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-24T17:24:49.400 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-psutil. 2024-04-24T17:24:49.427 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-24T17:24:49.466 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-24T17:24:49.710 INFO:teuthology.orchestra.run.smithi032.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-24T17:24:49.738 INFO:teuthology.orchestra.run.smithi032.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-24T17:24:49.803 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-24T17:24:49.819 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-py. 2024-04-24T17:24:49.847 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-24T17:24:49.894 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-24T17:24:49.903 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-24T17:24:50.214 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-pygments. 2024-04-24T17:24:50.241 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-24T17:24:50.271 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-24T17:24:50.280 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-24T17:24:50.397 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-24T17:24:50.665 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-24T17:24:50.675 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-24T17:24:50.691 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-24T17:24:50.734 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-24T17:24:50.941 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-24T17:24:51.036 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-toml. 2024-04-24T17:24:51.064 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-24T17:24:51.103 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-24T17:24:51.242 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up zip (3.0-12build2) ... 2024-04-24T17:24:51.368 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-24T17:24:51.439 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-pytest. 2024-04-24T17:24:51.466 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-24T17:24:51.505 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-24T17:24:51.825 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-simplejson. 2024-04-24T17:24:51.841 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-24T17:24:51.875 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-24T17:24:52.062 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-24T17:24:52.144 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-24T17:24:52.162 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-24T17:24:52.194 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-24T17:24:52.313 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-24T17:24:52.439 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-24T17:24:52.565 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-24T17:24:52.831 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package radosgw. 2024-04-24T17:24:52.859 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../116-radosgw_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:24:52.866 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-24T17:24:52.897 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking radosgw (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:53.276 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-24T17:24:53.518 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-24T17:24:53.684 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package rbd-fuse. 2024-04-24T17:24:53.712 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-3285-g5904d294-1jammy_amd64.deb ... 2024-04-24T17:24:53.758 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking rbd-fuse (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:53.895 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-24T17:24:54.022 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-24T17:24:54.188 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-24T17:24:54.272 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-24T17:24:54.406 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-24T17:24:54.539 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-24T17:24:54.649 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-24T17:24:54.798 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-24T17:24:54.924 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-ceph-common (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:54.995 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-24T17:24:55.217 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-idna (3.3-1) ... 2024-04-24T17:24:55.450 INFO:teuthology.orchestra.run.smithi082.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-24T17:24:55.515 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up cephadm (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:24:55.544 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-24T17:24:55.998 INFO:teuthology.orchestra.run.smithi082.stdout:Adding system user cephadm....done 2024-04-24T17:24:56.146 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-24T17:24:56.431 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-24T17:24:56.674 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-24T17:24:56.732 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-24T17:24:56.792 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-24T17:24:56.976 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T17:24:57.051 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-24T17:24:57.094 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-24T17:24:57.327 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-24T17:24:57.336 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-24T17:24:57.453 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-24T17:24:57.454 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-24T17:24:57.580 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-24T17:24:57.755 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-24T17:24:57.873 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-24T17:24:58.115 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-24T17:24:58.132 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-24T17:24:58.366 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-24T17:24:58.392 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T17:24:58.518 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-24T17:24:58.634 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-24T17:24:58.828 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-24T17:24:59.014 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-24T17:24:59.079 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-24T17:24:59.346 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-24T17:24:59.361 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-24T17:24:59.606 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-24T17:24:59.621 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-24T17:24:59.856 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-24T17:24:59.880 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-ceph-argparse (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:00.124 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-24T17:25:00.139 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-24T17:25:00.240 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-24T17:25:00.242 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-24T17:25:00.341 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-24T17:25:00.476 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-24T17:25:00.510 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-24T17:25:00.627 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-24T17:25:00.735 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-24T17:25:00.929 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-24T17:25:01.206 INFO:teuthology.orchestra.run.smithi082.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-24T17:25:01.263 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-24T17:25:01.432 INFO:teuthology.orchestra.run.smithi082.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-24T17:25:01.496 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-24T17:25:01.522 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-24T17:25:01.622 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-24T17:25:01.673 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-24T17:25:01.799 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-24T17:25:01.881 INFO:teuthology.orchestra.run.smithi032.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-24T17:25:01.973 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-24T17:25:02.074 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-24T17:25:02.143 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-24T17:25:02.269 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-24T17:25:02.334 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-24T17:25:02.585 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-24T17:25:02.662 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-24T17:25:02.894 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up zip (3.0-12build2) ... 2024-04-24T17:25:02.930 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-24T17:25:03.020 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-24T17:25:03.056 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-24T17:25:03.315 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T17:25:03.441 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-24T17:25:03.672 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-24T17:25:03.767 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-24T17:25:03.914 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-24T17:25:04.040 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-24T17:25:04.077 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-24T17:25:04.150 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-24T17:25:04.345 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-24T17:25:04.434 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-24T17:25:04.470 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-24T17:25:04.738 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-24T17:25:04.802 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-24T17:25:05.014 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-24T17:25:05.053 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-24T17:25:05.367 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-24T17:25:05.454 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-24T17:25:05.747 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-24T17:25:05.761 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-24T17:25:06.037 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-24T17:25:06.082 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-24T17:25:06.333 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-24T17:25:06.415 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-24T17:25:06.459 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-ceph-common (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:06.541 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up librados2 (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:06.667 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libsqlite3-mod-ceph (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:06.777 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-idna (3.3-1) ... 2024-04-24T17:25:06.876 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-24T17:25:07.103 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-24T17:25:07.179 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libcephfs2 (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:07.305 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libradosstriper1 (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:07.431 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-24T17:25:07.708 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up librbd1 (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:07.809 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-mgr-modules-core (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:07.910 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-fuse (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:08.122 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-24T17:25:08.122 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-24T17:25:08.225 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-24T17:25:08.476 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T17:25:08.611 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-24T17:25:08.612 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up libcephfs-dev (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:08.737 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-rados (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:08.863 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up librgw2 (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:08.870 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-24T17:25:08.989 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-24T17:25:08.996 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-24T17:25:09.122 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-24T17:25:09.415 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-24T17:25:09.666 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-24T17:25:09.917 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T17:25:10.043 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-24T17:25:10.243 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-rbd (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:10.336 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-24T17:25:10.385 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up rbd-fuse (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:10.487 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-rgw (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:10.596 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up python3-cephfs (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:10.612 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-24T17:25:10.722 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-common (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:10.879 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-24T17:25:11.130 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-24T17:25:11.265 INFO:teuthology.orchestra.run.smithi032.stdout:Adding group ceph....done 2024-04-24T17:25:11.373 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-24T17:25:11.633 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-24T17:25:11.633 INFO:teuthology.orchestra.run.smithi032.stdout:Adding system user ceph....done 2024-04-24T17:25:11.758 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-24T17:25:11.802 INFO:teuthology.orchestra.run.smithi032.stdout:Setting system user ceph properties....done 2024-04-24T17:25:11.820 INFO:teuthology.orchestra.run.smithi032.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-24T17:25:11.965 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-24T17:25:12.034 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-24T17:25:12.144 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-24T17:25:12.310 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-24T17:25:12.428 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-24T17:25:12.729 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-24T17:25:12.806 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-test (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:12.949 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up radosgw (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:12.980 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-24T17:25:13.081 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-24T17:25:13.199 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-24T17:25:13.288 INFO:teuthology.orchestra.run.smithi082.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-24T17:25:13.526 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-24T17:25:13.635 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-24T17:25:13.763 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-24T17:25:13.763 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-24T17:25:14.036 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-24T17:25:14.243 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-base (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:14.304 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-24T17:25:14.413 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-24T17:25:14.587 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-24T17:25:14.672 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T17:25:14.798 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-24T17:25:15.133 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-24T17:25:15.196 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-mds (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:15.434 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-24T17:25:15.438 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-24T17:25:15.438 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-24T17:25:15.710 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-24T17:25:15.836 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-24T17:25:15.888 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-mgr (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:16.103 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-24T17:25:16.115 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-24T17:25:16.115 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-24T17:25:16.379 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-24T17:25:16.606 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-osd (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:16.708 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-24T17:25:16.885 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-24T17:25:16.885 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-24T17:25:17.093 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-24T17:25:17.369 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-24T17:25:17.408 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-mgr-k8sevents (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:17.534 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:17.685 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-mon (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:17.696 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-24T17:25:17.797 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up librados2 (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:17.891 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-24T17:25:17.891 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-24T17:25:17.907 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libsqlite3-mod-ceph (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:18.033 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-24T17:25:18.394 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libcephfs2 (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:18.420 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-mgr-cephadm (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:18.520 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libradosstriper1 (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:18.571 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:18.663 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-24T17:25:18.697 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-mgr-dashboard (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:18.848 INFO:teuthology.orchestra.run.smithi032.stdout:Setting up ceph-volume (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:18.922 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up librbd1 (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:19.032 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-mgr-modules-core (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:19.133 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-fuse (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:19.192 INFO:teuthology.orchestra.run.smithi032.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-24T17:25:19.310 INFO:teuthology.orchestra.run.smithi032.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-24T17:25:19.333 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-24T17:25:19.333 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-24T17:25:19.462 INFO:teuthology.orchestra.run.smithi032.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-24T17:25:19.826 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libcephfs-dev (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:19.952 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-rados (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:20.077 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up librgw2 (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:20.203 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-24T17:25:20.555 INFO:teuthology.orchestra.run.smithi032.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-24T17:25:21.209 INFO:teuthology.orchestra.run.smithi032.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-24T17:25:21.214 DEBUG:teuthology.parallel:result is None 2024-04-24T17:25:21.432 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-rbd (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:21.533 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up rbd-fuse (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:21.650 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-rgw (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:21.776 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-cephfs (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:21.927 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-common (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:22.479 INFO:teuthology.orchestra.run.smithi082.stdout:Adding group ceph....done 2024-04-24T17:25:22.863 INFO:teuthology.orchestra.run.smithi082.stdout:Adding system user ceph....done 2024-04-24T17:25:23.031 INFO:teuthology.orchestra.run.smithi082.stdout:Setting system user ceph properties....done 2024-04-24T17:25:23.050 INFO:teuthology.orchestra.run.smithi082.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-24T17:25:23.188 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-24T17:25:23.529 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-24T17:25:24.028 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-test (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:24.154 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up radosgw (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:24.628 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-24T17:25:24.629 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-24T17:25:25.283 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-base (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:25.627 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-24T17:25:26.160 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-mds (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:26.407 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-24T17:25:26.407 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-24T17:25:26.939 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-mgr (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:27.206 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-24T17:25:27.206 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-24T17:25:27.690 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-osd (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:27.963 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-24T17:25:27.963 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-24T17:25:28.483 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-mgr-k8sevents (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:28.610 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:28.761 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-mon (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:28.968 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-24T17:25:28.968 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-24T17:25:29.487 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-mgr-cephadm (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:29.655 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:29.780 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-mgr-dashboard (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:29.923 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-volume (19.0.0-3285-g5904d294-1jammy) ... 2024-04-24T17:25:30.268 INFO:teuthology.orchestra.run.smithi082.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-24T17:25:30.369 INFO:teuthology.orchestra.run.smithi082.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-24T17:25:30.521 INFO:teuthology.orchestra.run.smithi082.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-24T17:25:31.204 INFO:teuthology.orchestra.run.smithi082.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-24T17:25:31.974 INFO:teuthology.orchestra.run.smithi082.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-24T17:25:31.978 DEBUG:teuthology.parallel:result is None 2024-04-24T17:25:31.979 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T17:25:32.126 DEBUG:teuthology.orchestra.run.smithi032:> dpkg-query -W -f '${Version}' ceph 2024-04-24T17:25:32.146 INFO:teuthology.orchestra.run.smithi032.stdout:19.0.0-3285-g5904d294-1jammy 2024-04-24T17:25:32.146 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3285-g5904d294-1jammy 2024-04-24T17:25:32.146 INFO:teuthology.task.install:The correct ceph version 19.0.0-3285-g5904d294-1jammy is installed. 2024-04-24T17:25:32.148 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T17:25:32.290 DEBUG:teuthology.orchestra.run.smithi082:> dpkg-query -W -f '${Version}' ceph 2024-04-24T17:25:32.309 INFO:teuthology.orchestra.run.smithi082.stdout:19.0.0-3285-g5904d294-1jammy 2024-04-24T17:25:32.309 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3285-g5904d294-1jammy 2024-04-24T17:25:32.309 INFO:teuthology.task.install:The correct ceph version 19.0.0-3285-g5904d294-1jammy is installed. 2024-04-24T17:25:32.310 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-24T17:25:32.310 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-24T17:25:32.311 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-24T17:25:32.326 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-04-24T17:25:32.326 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-24T17:25:32.362 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-24T17:25:32.363 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-24T17:25:32.363 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/daemon-helper 2024-04-24T17:25:32.385 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-24T17:25:32.440 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-04-24T17:25:32.440 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/daemon-helper 2024-04-24T17:25:32.453 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-24T17:25:32.506 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-24T17:25:32.506 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-24T17:25:32.506 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-24T17:25:32.520 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-24T17:25:32.575 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-04-24T17:25:32.576 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-24T17:25:32.588 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-24T17:25:32.642 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-24T17:25:32.642 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-24T17:25:32.642 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/stdin-killer 2024-04-24T17:25:32.655 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-24T17:25:32.708 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-04-24T17:25:32.709 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/stdin-killer 2024-04-24T17:25:32.721 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-24T17:25:32.773 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-24T17:25:32.868 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr2': False, 'ms type': 'async'}, 'mon': {'debug mgrc': 20, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd class default list': '*', 'osd class load list': '*', 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'crush_tunables': 'jewel', 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'reached quota', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)'], 'log-only-match': ['CEPHADM_'], 'mon_bind_msgr2': False, 'sha1': '5904d29475f5be602879d9fb26280e89b808d5cc', 'cephadm_mode': 'root'} 2024-04-24T17:25:32.868 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T17:25:32.868 INFO:tasks.cephadm:Cluster fsid is a7981b84-025f-11ef-bc93-c7b262605968 2024-04-24T17:25:32.868 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-24T17:25:32.869 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '[v1:172.21.15.32:6789]', 'mon.c': '[v1:172.21.15.32:6790]', 'mon.b': '[v1:172.21.15.82:6789]'} 2024-04-24T17:25:32.869 INFO:tasks.cephadm:First mon is mon.a on smithi032 2024-04-24T17:25:32.869 INFO:tasks.cephadm:First mgr is y 2024-04-24T17:25:32.869 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-24T17:25:32.869 DEBUG:teuthology.orchestra.run.smithi032:> sudo hostname $(hostname -s) 2024-04-24T17:25:32.884 DEBUG:teuthology.orchestra.run.smithi082:> sudo hostname $(hostname -s) 2024-04-24T17:25:32.899 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-24T17:25:32.899 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T17:25:33.040 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '5904d29475f5be602879d9fb26280e89b808d5cc', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=jammy,DIST=jammy,MACHINE_SIZE=gigantic/78851/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3285-g5904d294', 'node_name': '172.21.2.13+braggi13', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=jammy,DIST=jammy,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3285-g5904d294-1jammy'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default/', 'distro_codename': 'jammy', 'modified': '2024-04-24 15:35:30.146514', 'distro_version': '22.04', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-adk-testing-2024-04-24-0937', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/flavors/default/', 'archs': ['x86_64'], 'distro': 'ubuntu'}] 2024-04-24T17:25:33.176 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-adk-testing-2024-04-24-0937, sha1 5904d29475f5be602879d9fb26280e89b808d5cc from https://shaman.ceph.com/api/search/?project=ceph&distros=ubuntu%2F22.04%2Fx86_64&flavor=default&sha1=5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T17:25:33.177 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/x86_64/flavors/default/cephadm 2024-04-24T17:25:33.177 DEBUG:teuthology.orchestra.run.smithi032:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-24T17:25:34.421 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-rw-r-- 1 ubuntu ubuntu 789946 Apr 24 17:25 /home/ubuntu/cephtest/cephadm 2024-04-24T17:25:34.421 DEBUG:teuthology.orchestra.run.smithi082:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-adk-testing-2024-04-24-0937/5904d29475f5be602879d9fb26280e89b808d5cc/ubuntu/jammy/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-24T17:25:35.416 INFO:teuthology.orchestra.run.smithi082.stdout:-rw-rw-r-- 1 ubuntu ubuntu 789946 Apr 24 17:25 /home/ubuntu/cephtest/cephadm 2024-04-24T17:25:35.416 DEBUG:teuthology.orchestra.run.smithi032:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-24T17:25:35.424 DEBUG:teuthology.orchestra.run.smithi082:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-24T17:25:35.443 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc on all hosts... 2024-04-24T17:25:35.444 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc pull 2024-04-24T17:25:35.470 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc pull 2024-04-24T17:25:35.640 INFO:teuthology.orchestra.run.smithi032.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc... 2024-04-24T17:25:35.640 INFO:teuthology.orchestra.run.smithi082.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc... 2024-04-24T17:26:50.090 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-04-24T17:26:50.090 INFO:teuthology.orchestra.run.smithi082.stdout: "ceph_version": "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)", 2024-04-24T17:26:50.090 INFO:teuthology.orchestra.run.smithi082.stdout: "image_id": "8bb2087275ff8b3edc22adaa46e72c303749ca6355b50780fa41551d025b2f8d", 2024-04-24T17:26:50.090 INFO:teuthology.orchestra.run.smithi082.stdout: "repo_digests": [ 2024-04-24T17:26:50.090 INFO:teuthology.orchestra.run.smithi082.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a" 2024-04-24T17:26:50.090 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-04-24T17:26:50.091 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-04-24T17:26:51.001 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-24T17:26:51.002 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_version": "ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev)", 2024-04-24T17:26:51.002 INFO:teuthology.orchestra.run.smithi032.stdout: "image_id": "8bb2087275ff8b3edc22adaa46e72c303749ca6355b50780fa41551d025b2f8d", 2024-04-24T17:26:51.002 INFO:teuthology.orchestra.run.smithi032.stdout: "repo_digests": [ 2024-04-24T17:26:51.002 INFO:teuthology.orchestra.run.smithi032.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a" 2024-04-24T17:26:51.002 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-24T17:26:51.002 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-24T17:26:51.043 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /etc/ceph 2024-04-24T17:26:51.057 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /etc/ceph 2024-04-24T17:26:51.071 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 777 /etc/ceph 2024-04-24T17:26:51.111 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 777 /etc/ceph 2024-04-24T17:26:51.125 INFO:tasks.cephadm:Writing seed config... 2024-04-24T17:26:51.126 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-24T17:26:51.126 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-24T17:26:51.126 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-24T17:26:51.126 INFO:tasks.cephadm: override: [client] debug objecter = 20 2024-04-24T17:26:51.126 INFO:tasks.cephadm: override: [client] debug rados = 20 2024-04-24T17:26:51.126 INFO:tasks.cephadm: override: [global] ms bind msgr2 = False 2024-04-24T17:26:51.126 INFO:tasks.cephadm: override: [global] ms type = async 2024-04-24T17:26:51.126 INFO:tasks.cephadm: override: [mon] debug mgrc = 20 2024-04-24T17:26:51.126 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-24T17:26:51.126 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-24T17:26:51.126 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-24T17:26:51.127 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-04-24T17:26:51.127 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-04-24T17:26:51.127 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-04-24T17:26:51.127 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-04-24T17:26:51.127 INFO:tasks.cephadm: override: [mon] mon warn on pool no app = False 2024-04-24T17:26:51.127 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-04-24T17:26:51.127 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-24T17:26:51.127 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-24T17:26:51.127 INFO:tasks.cephadm: override: [osd] osd class default list = * 2024-04-24T17:26:51.127 INFO:tasks.cephadm: override: [osd] osd class load list = * 2024-04-24T17:26:51.127 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-04-24T17:26:51.127 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-04-24T17:26:51.127 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-04-24T17:26:51.127 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-04-24T17:26:51.127 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-04-24T17:26:51.128 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-24T17:26:51.128 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-04-24T17:26:51.128 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-24T17:26:51.128 DEBUG:teuthology.orchestra.run.smithi032:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-24T17:26:51.160 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = a7981b84-025f-11ef-bc93-c7b262605968 ms bind msgr2 = False ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd class default list = * osd class load list = * osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mgrc = 20 debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 mon warn on pool no app = False paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] debug ms = 1 debug objecter = 20 debug rados = 20 2024-04-24T17:26:51.161 DEBUG:teuthology.orchestra.run.smithi032:mon.a> sudo journalctl -f -n 0 -u ceph-a7981b84-025f-11ef-bc93-c7b262605968@mon.a.service 2024-04-24T17:26:51.205 DEBUG:teuthology.orchestra.run.smithi032:mgr.y> sudo journalctl -f -n 0 -u ceph-a7981b84-025f-11ef-bc93-c7b262605968@mgr.y.service 2024-04-24T17:26:51.248 INFO:tasks.cephadm:Bootstrapping... 2024-04-24T17:26:51.248 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc -v bootstrap --fsid a7981b84-025f-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-addrv '[v1:172.21.15.32:6789]' --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-24T17:26:51.463 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-24T17:26:51.463 INFO:teuthology.orchestra.run.smithi032.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc', '-v', 'bootstrap', '--fsid', 'a7981b84-025f-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-addrv', '[v1:172.21.15.32:6789]', '--skip-admin-label'] 2024-04-24T17:26:51.463 INFO:teuthology.orchestra.run.smithi032.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-24T17:26:51.464 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying podman|docker is present... 2024-04-24T17:26:51.464 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying lvm2 is present... 2024-04-24T17:26:51.464 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying time synchronization is in place... 2024-04-24T17:26:51.469 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-24T17:26:51.470 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-24T17:26:51.475 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-24T17:26:51.475 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout inactive 2024-04-24T17:26:51.481 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl is-enabled chronyd.service 2024-04-24T17:26:51.481 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Failed to get unit file state for chronyd.service: No such file or directory 2024-04-24T17:26:51.485 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 3 from systemctl is-active chronyd.service 2024-04-24T17:26:51.485 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout inactive 2024-04-24T17:26:51.489 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl is-enabled systemd-timesyncd.service 2024-04-24T17:26:51.489 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout masked 2024-04-24T17:26:51.492 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 3 from systemctl is-active systemd-timesyncd.service 2024-04-24T17:26:51.492 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout inactive 2024-04-24T17:26:51.496 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl is-enabled ntpd.service 2024-04-24T17:26:51.496 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Failed to get unit file state for ntpd.service: No such file or directory 2024-04-24T17:26:51.500 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 3 from systemctl is-active ntpd.service 2024-04-24T17:26:51.500 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout inactive 2024-04-24T17:26:51.504 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout enabled 2024-04-24T17:26:51.508 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout active 2024-04-24T17:26:51.508 INFO:teuthology.orchestra.run.smithi032.stdout:Unit ntp.service is enabled and running 2024-04-24T17:26:51.508 INFO:teuthology.orchestra.run.smithi032.stdout:Repeating the final host check... 2024-04-24T17:26:51.508 INFO:teuthology.orchestra.run.smithi032.stdout:docker (/usr/bin/docker) is present 2024-04-24T17:26:51.509 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl is present 2024-04-24T17:26:51.509 INFO:teuthology.orchestra.run.smithi032.stdout:lvcreate is present 2024-04-24T17:26:51.512 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-24T17:26:51.512 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-24T17:26:51.515 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-24T17:26:51.516 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout inactive 2024-04-24T17:26:51.519 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl is-enabled chronyd.service 2024-04-24T17:26:51.519 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Failed to get unit file state for chronyd.service: No such file or directory 2024-04-24T17:26:51.522 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 3 from systemctl is-active chronyd.service 2024-04-24T17:26:51.523 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout inactive 2024-04-24T17:26:51.526 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl is-enabled systemd-timesyncd.service 2024-04-24T17:26:51.527 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout masked 2024-04-24T17:26:51.530 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 3 from systemctl is-active systemd-timesyncd.service 2024-04-24T17:26:51.530 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout inactive 2024-04-24T17:26:51.533 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl is-enabled ntpd.service 2024-04-24T17:26:51.533 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Failed to get unit file state for ntpd.service: No such file or directory 2024-04-24T17:26:51.537 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 3 from systemctl is-active ntpd.service 2024-04-24T17:26:51.537 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout inactive 2024-04-24T17:26:51.541 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout enabled 2024-04-24T17:26:51.545 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout active 2024-04-24T17:26:51.545 INFO:teuthology.orchestra.run.smithi032.stdout:Unit ntp.service is enabled and running 2024-04-24T17:26:51.545 INFO:teuthology.orchestra.run.smithi032.stdout:Host looks OK 2024-04-24T17:26:51.545 INFO:teuthology.orchestra.run.smithi032.stdout:Cluster fsid: a7981b84-025f-11ef-bc93-c7b262605968 2024-04-24T17:26:51.546 INFO:teuthology.orchestra.run.smithi032.stdout:Acquiring lock 140085611620256 on /run/cephadm/a7981b84-025f-11ef-bc93-c7b262605968.lock 2024-04-24T17:26:51.546 INFO:teuthology.orchestra.run.smithi032.stdout:Lock 140085611620256 acquired on /run/cephadm/a7981b84-025f-11ef-bc93-c7b262605968.lock 2024-04-24T17:26:51.546 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying IP 172.21.15.32 port 6789 ... 2024-04-24T17:26:51.546 INFO:teuthology.orchestra.run.smithi032.stdout:Base mon IP(s) is [172.21.15.32:6789], mon addrv is [v1:172.21.15.32:6789] 2024-04-24T17:26:51.548 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 2024-04-24T17:26:51.548 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/sbin/ip: stdout 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 2024-04-24T17:26:51.548 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.32 2024-04-24T17:26:51.550 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-24T17:26:51.550 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-04-24T17:26:51.550 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1739sec hoplimit 64 pref medium 2024-04-24T17:26:51.552 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-24T17:26:51.553 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-24T17:26:51.553 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-24T17:26:51.553 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-24T17:26:51.553 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/sbin/ip: stdout inet6 fe80::ae1f:6bff:fef8:2602/64 scope link 2024-04-24T17:26:51.553 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-24T17:26:51.553 INFO:teuthology.orchestra.run.smithi032.stdout:Mon IP `172.21.15.32` is in CIDR network `172.21.0.0/20` 2024-04-24T17:26:51.554 INFO:teuthology.orchestra.run.smithi032.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20'] 2024-04-24T17:26:51.554 INFO:teuthology.orchestra.run.smithi032.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-24T17:26:51.554 INFO:teuthology.orchestra.run.smithi032.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc... 2024-04-24T17:26:51.931 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/docker: stdout 5904d29475f5be602879d9fb26280e89b808d5cc: Pulling from ceph-ci/ceph 2024-04-24T17:26:51.931 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/docker: stdout Digest: sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a 2024-04-24T17:26:51.931 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/docker: stdout Status: Image is up to date for quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T17:26:51.931 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/docker: stdout quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc 2024-04-24T17:26:53.414 INFO:teuthology.orchestra.run.smithi032.stdout:ceph: stdout ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev) 2024-04-24T17:26:53.414 INFO:teuthology.orchestra.run.smithi032.stdout:Ceph version: ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev) 2024-04-24T17:26:53.414 INFO:teuthology.orchestra.run.smithi032.stdout:Extracting ceph user uid/gid from container image... 2024-04-24T17:26:54.735 INFO:teuthology.orchestra.run.smithi032.stdout:stat: stdout 167 167 2024-04-24T17:26:54.735 INFO:teuthology.orchestra.run.smithi032.stdout:Creating initial keys... 2024-04-24T17:26:56.083 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-authtool: stdout AQDfQClmLiN1HhAAvRTnPSYfdy2rgGkLFhPO1g== 2024-04-24T17:26:57.370 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-authtool: stdout AQDgQClmEMeMMBAAxthi1UDaED1B9k6Rmlx12A== 2024-04-24T17:26:58.693 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-authtool: stdout AQDiQClmF4/aBxAAanZy4vW8MUqyi3H5Xm1Nmw== 2024-04-24T17:26:58.693 INFO:teuthology.orchestra.run.smithi032.stdout:Creating initial monmap... 2024-04-24T17:27:00.148 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-24T17:27:00.148 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-24T17:27:00.148 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to a7981b84-025f-11ef-bc93-c7b262605968 2024-04-24T17:27:00.149 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-24T17:27:00.149 INFO:teuthology.orchestra.run.smithi032.stdout:monmaptool for a [v1:172.21.15.32:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-24T17:27:00.149 INFO:teuthology.orchestra.run.smithi032.stdout:setting min_mon_release = quincy 2024-04-24T17:27:00.149 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/monmaptool: set fsid to a7981b84-025f-11ef-bc93-c7b262605968 2024-04-24T17:27:00.149 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-24T17:27:00.149 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T17:27:00.149 INFO:teuthology.orchestra.run.smithi032.stdout:Creating mon... 2024-04-24T17:27:01.605 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.859+0000 7f7d0ee60c80 0 set uid:gid to 167:167 (ceph:ceph) 2024-04-24T17:27:01.605 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.859+0000 7f7d0ee60c80 1 imported monmap: 2024-04-24T17:27:01.605 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr epoch 0 2024-04-24T17:27:01.605 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr fsid a7981b84-025f-11ef-bc93-c7b262605968 2024-04-24T17:27:01.605 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr last_changed 2024-04-24T17:26:59.605652+0000 2024-04-24T17:27:01.605 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr created 2024-04-24T17:26:59.605652+0000 2024-04-24T17:27:01.607 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr min_mon_release 17 (quincy) 2024-04-24T17:27:01.607 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr election_strategy: 1 2024-04-24T17:27:01.607 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr 0: v1:172.21.15.32:6789/0 mon.a 2024-04-24T17:27:01.607 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr 2024-04-24T17:27:01.607 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.859+0000 7f7d0ee60c80 0 /usr/bin/ceph-mon: set fsid to a7981b84-025f-11ef-bc93-c7b262605968 2024-04-24T17:27:01.607 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: RocksDB version: 7.9.2 2024-04-24T17:27:01.607 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr 2024-04-24T17:27:01.607 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Git sha 0 2024-04-24T17:27:01.608 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Compile date 2024-04-24 13:56:12 2024-04-24T17:27:01.608 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: DB SUMMARY 2024-04-24T17:27:01.608 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr 2024-04-24T17:27:01.608 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: DB Session ID: YHKX419WXEVA1COTNBO2 2024-04-24T17:27:01.608 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr 2024-04-24T17:27:01.608 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 0, files: 2024-04-24T17:27:01.610 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr 2024-04-24T17:27:01.610 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 2024-04-24T17:27:01.610 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr 2024-04-24T17:27:01.610 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.error_if_exists: 0 2024-04-24T17:27:01.610 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.create_if_missing: 1 2024-04-24T17:27:01.611 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.paranoid_checks: 1 2024-04-24T17:27:01.611 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.flush_verify_memtable_count: 1 2024-04-24T17:27:01.611 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-24T17:27:01.611 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-24T17:27:01.611 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.env: 0x555d0488acc0 2024-04-24T17:27:01.611 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.fs: PosixFileSystem 2024-04-24T17:27:01.611 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.info_log: 0x555d0629b020 2024-04-24T17:27:01.611 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.max_file_opening_threads: 16 2024-04-24T17:27:01.611 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.statistics: (nil) 2024-04-24T17:27:01.611 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.use_fsync: 0 2024-04-24T17:27:01.611 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.max_log_file_size: 0 2024-04-24T17:27:01.611 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-24T17:27:01.611 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.log_file_time_to_roll: 0 2024-04-24T17:27:01.612 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.keep_log_file_num: 1000 2024-04-24T17:27:01.612 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.recycle_log_file_num: 0 2024-04-24T17:27:01.612 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.allow_fallocate: 1 2024-04-24T17:27:01.615 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.allow_mmap_reads: 0 2024-04-24T17:27:01.615 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.allow_mmap_writes: 0 2024-04-24T17:27:01.615 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.use_direct_reads: 0 2024-04-24T17:27:01.615 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-24T17:27:01.615 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.create_missing_column_families: 0 2024-04-24T17:27:01.615 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.db_log_dir: 2024-04-24T17:27:01.615 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.wal_dir: 2024-04-24T17:27:01.615 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.table_cache_numshardbits: 6 2024-04-24T17:27:01.615 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-04-24T17:27:01.616 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-04-24T17:27:01.616 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-24T17:27:01.616 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-24T17:27:01.616 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-04-24T17:27:01.616 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.advise_random_on_open: 1 2024-04-24T17:27:01.616 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.db_write_buffer_size: 0 2024-04-24T17:27:01.616 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.write_buffer_manager: 0x555d0626aaa0 2024-04-24T17:27:01.616 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-24T17:27:01.616 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-24T17:27:01.616 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.use_adaptive_mutex: 0 2024-04-24T17:27:01.616 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.rate_limiter: (nil) 2024-04-24T17:27:01.616 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-24T17:27:01.616 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.wal_recovery_mode: 2 2024-04-24T17:27:01.616 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.enable_thread_tracking: 0 2024-04-24T17:27:01.616 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.enable_pipelined_write: 0 2024-04-24T17:27:01.617 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.unordered_write: 0 2024-04-24T17:27:01.617 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-24T17:27:01.617 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-24T17:27:01.617 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-24T17:27:01.617 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-24T17:27:01.617 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.row_cache: None 2024-04-24T17:27:01.620 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.wal_filter: None 2024-04-24T17:27:01.620 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-24T17:27:01.621 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.allow_ingest_behind: 0 2024-04-24T17:27:01.621 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.two_write_queues: 0 2024-04-24T17:27:01.621 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.manual_wal_flush: 0 2024-04-24T17:27:01.621 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.wal_compression: 0 2024-04-24T17:27:01.621 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.atomic_flush: 0 2024-04-24T17:27:01.621 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-24T17:27:01.621 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.persist_stats_to_disk: 0 2024-04-24T17:27:01.621 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-04-24T17:27:01.621 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.log_readahead_size: 0 2024-04-24T17:27:01.621 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-24T17:27:01.621 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.best_efforts_recovery: 0 2024-04-24T17:27:01.621 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-24T17:27:01.621 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-24T17:27:01.622 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.allow_data_in_errors: 0 2024-04-24T17:27:01.622 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.db_host_id: __hostname__ 2024-04-24T17:27:01.622 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.enforce_single_del_contracts: true 2024-04-24T17:27:01.622 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.max_background_jobs: 2 2024-04-24T17:27:01.622 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.max_background_compactions: -1 2024-04-24T17:27:01.622 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.max_subcompactions: 1 2024-04-24T17:27:01.622 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-24T17:27:01.622 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-24T17:27:01.622 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.delayed_write_rate : 16777216 2024-04-24T17:27:01.622 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.max_total_wal_size: 0 2024-04-24T17:27:01.622 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-24T17:27:01.622 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.stats_dump_period_sec: 600 2024-04-24T17:27:01.622 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.stats_persist_period_sec: 600 2024-04-24T17:27:01.622 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-24T17:27:01.623 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.max_open_files: -1 2024-04-24T17:27:01.623 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.bytes_per_sync: 0 2024-04-24T17:27:01.623 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-04-24T17:27:01.623 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-04-24T17:27:01.623 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.compaction_readahead_size: 0 2024-04-24T17:27:01.623 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.max_background_flushes: -1 2024-04-24T17:27:01.623 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Compression algorithms supported: 2024-04-24T17:27:01.627 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-24T17:27:01.628 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: kZSTD supported: 0 2024-04-24T17:27:01.628 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: kXpressCompression supported: 0 2024-04-24T17:27:01.628 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: kLZ4HCCompression supported: 1 2024-04-24T17:27:01.628 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: kZlibCompression supported: 1 2024-04-24T17:27:01.628 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: kSnappyCompression supported: 1 2024-04-24T17:27:01.628 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: kLZ4Compression supported: 1 2024-04-24T17:27:01.628 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: kBZip2Compression supported: 0 2024-04-24T17:27:01.628 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-04-24T17:27:01.628 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: DMutex implementation: pthread_mutex_t 2024-04-24T17:27:01.628 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: [db/db_impl/db_impl_open.cc:317] Creating manifest 1 2024-04-24T17:27:01.628 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr 2024-04-24T17:27:01.628 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 2024-04-24T17:27:01.628 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr 2024-04-24T17:27:01.628 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-24T17:27:01.629 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr 2024-04-24T17:27:01.629 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-24T17:27:01.629 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.merge_operator: 2024-04-24T17:27:01.629 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.compaction_filter: None 2024-04-24T17:27:01.629 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.compaction_filter_factory: None 2024-04-24T17:27:01.629 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.sst_partitioner_factory: None 2024-04-24T17:27:01.629 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-04-24T17:27:01.629 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.table_factory: BlockBasedTable 2024-04-24T17:27:01.629 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x555d0629b1e0) 2024-04-24T17:27:01.629 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr cache_index_and_filter_blocks: 1 2024-04-24T17:27:01.629 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr cache_index_and_filter_blocks_with_high_priority: 0 2024-04-24T17:27:01.629 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-24T17:27:01.629 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr pin_top_level_index_and_filter: 1 2024-04-24T17:27:01.629 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr index_type: 0 2024-04-24T17:27:01.629 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr data_block_index_type: 0 2024-04-24T17:27:01.630 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr index_shortening: 1 2024-04-24T17:27:01.630 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr data_block_hash_table_util_ratio: 0.750000 2024-04-24T17:27:01.630 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr checksum: 4 2024-04-24T17:27:01.630 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr no_block_cache: 0 2024-04-24T17:27:01.630 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr block_cache: 0x555d0629d1f0 2024-04-24T17:27:01.630 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr block_cache_name: BinnedLRUCache 2024-04-24T17:27:01.630 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr block_cache_options: 2024-04-24T17:27:01.630 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr capacity : 536870912 2024-04-24T17:27:01.630 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr num_shard_bits : 4 2024-04-24T17:27:01.630 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr strict_capacity_limit : 0 2024-04-24T17:27:01.630 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr high_pri_pool_ratio: 0.000 2024-04-24T17:27:01.630 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr block_cache_compressed: (nil) 2024-04-24T17:27:01.630 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr persistent_cache: (nil) 2024-04-24T17:27:01.630 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr block_size: 4096 2024-04-24T17:27:01.631 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr block_size_deviation: 10 2024-04-24T17:27:01.631 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr block_restart_interval: 16 2024-04-24T17:27:01.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr index_block_restart_interval: 1 2024-04-24T17:27:01.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr metadata_block_size: 4096 2024-04-24T17:27:01.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr partition_filters: 0 2024-04-24T17:27:01.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr use_delta_encoding: 1 2024-04-24T17:27:01.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr filter_policy: bloomfilter 2024-04-24T17:27:01.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr whole_key_filtering: 1 2024-04-24T17:27:01.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr verify_compression: 0 2024-04-24T17:27:01.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr read_amp_bytes_per_bit: 0 2024-04-24T17:27:01.637 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr format_version: 5 2024-04-24T17:27:01.637 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr enable_index_compression: 1 2024-04-24T17:27:01.637 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr block_align: 0 2024-04-24T17:27:01.637 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr max_auto_readahead_size: 262144 2024-04-24T17:27:01.637 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr prepopulate_block_cache: 0 2024-04-24T17:27:01.637 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr initial_auto_readahead_size: 8192 2024-04-24T17:27:01.637 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr num_file_reads_for_auto_readahead: 2 2024-04-24T17:27:01.637 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr 2024-04-24T17:27:01.637 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.write_buffer_size: 33554432 2024-04-24T17:27:01.637 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.max_write_buffer_number: 2 2024-04-24T17:27:01.637 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.compression: NoCompression 2024-04-24T17:27:01.637 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.bottommost_compression: Disabled 2024-04-24T17:27:01.637 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.prefix_extractor: nullptr 2024-04-24T17:27:01.637 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-24T17:27:01.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.num_levels: 7 2024-04-24T17:27:01.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-24T17:27:01.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-24T17:27:01.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-24T17:27:01.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-24T17:27:01.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-24T17:27:01.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-24T17:27:01.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-24T17:27:01.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-24T17:27:01.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-24T17:27:01.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-24T17:27:01.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-24T17:27:01.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-24T17:27:01.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.compression_opts.window_bits: -14 2024-04-24T17:27:01.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.compression_opts.level: 32767 2024-04-24T17:27:01.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.compression_opts.strategy: 0 2024-04-24T17:27:01.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-24T17:27:01.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-24T17:27:01.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-24T17:27:01.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-24T17:27:01.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.compression_opts.enabled: false 2024-04-24T17:27:01.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-24T17:27:01.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-24T17:27:01.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-24T17:27:01.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-24T17:27:01.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.target_file_size_base: 67108864 2024-04-24T17:27:01.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.target_file_size_multiplier: 1 2024-04-24T17:27:01.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-24T17:27:01.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-24T17:27:01.640 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-24T17:27:01.640 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-24T17:27:01.640 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-24T17:27:01.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-24T17:27:01.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-24T17:27:01.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-24T17:27:01.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-24T17:27:01.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-24T17:27:01.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-24T17:27:01.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-24T17:27:01.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-24T17:27:01.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.arena_block_size: 1048576 2024-04-24T17:27:01.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-24T17:27:01.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-24T17:27:01.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.disable_auto_compactions: 0 2024-04-24T17:27:01.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-24T17:27:01.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-24T17:27:01.647 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-24T17:27:01.647 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-24T17:27:01.647 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-24T17:27:01.647 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-24T17:27:01.647 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-24T17:27:01.647 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-24T17:27:01.647 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-24T17:27:01.647 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-24T17:27:01.647 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-24T17:27:01.647 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.inplace_update_support: 0 2024-04-24T17:27:01.647 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-04-24T17:27:01.647 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-24T17:27:01.647 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-24T17:27:01.647 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.memtable_huge_page_size: 0 2024-04-24T17:27:01.648 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.bloom_locality: 0 2024-04-24T17:27:01.648 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.max_successive_merges: 0 2024-04-24T17:27:01.648 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-04-24T17:27:01.648 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.paranoid_file_checks: 0 2024-04-24T17:27:01.648 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.force_consistency_checks: 1 2024-04-24T17:27:01.648 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.report_bg_io_stats: 0 2024-04-24T17:27:01.648 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.ttl: 2592000 2024-04-24T17:27:01.648 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-04-24T17:27:01.648 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-24T17:27:01.648 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-24T17:27:01.648 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.enable_blob_files: false 2024-04-24T17:27:01.648 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.min_blob_size: 0 2024-04-24T17:27:01.648 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.blob_file_size: 268435456 2024-04-24T17:27:01.648 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.blob_compression_type: NoCompression 2024-04-24T17:27:01.649 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.enable_blob_garbage_collection: false 2024-04-24T17:27:01.649 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-24T17:27:01.649 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-24T17:27:01.649 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-24T17:27:01.649 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.blob_file_starting_level: 0 2024-04-24T17:27:01.649 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-24T17:27:01.649 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-24T17:27:01.649 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr 2024-04-24T17:27:01.649 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-24T17:27:01.649 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr 2024-04-24T17:27:01.649 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 550a0adb-8b63-4cbc-9c5c-892975aaeb49 2024-04-24T17:27:01.649 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr 2024-04-24T17:27:01.649 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.863+0000 7f7d0ee60c80 4 rocksdb: [db/version_set.cc:5047] Creating manifest 5 2024-04-24T17:27:01.649 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr 2024-04-24T17:27:01.649 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.867+0000 7f7d0ee60c80 4 rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x555d06394000 2024-04-24T17:27:01.650 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.867+0000 7f7d0ee60c80 4 rocksdb: DB pointer 0x555d06384000 2024-04-24T17:27:01.650 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.867+0000 7f7cfefc0700 4 rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-24T17:27:01.650 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.867+0000 7f7cfefc0700 4 rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-24T17:27:01.653 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr ** DB Stats ** 2024-04-24T17:27:01.653 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr Uptime(secs): 0.0 total, 0.0 interval 2024-04-24T17:27:01.654 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-24T17:27:01.654 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-24T17:27:01.654 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-24T17:27:01.654 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-24T17:27:01.654 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-24T17:27:01.654 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-24T17:27:01.654 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr 2024-04-24T17:27:01.654 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr ** Compaction Stats [default] ** 2024-04-24T17:27:01.654 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-24T17:27:01.654 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-24T17:27:01.654 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-24T17:27:01.654 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-24T17:27:01.654 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr 2024-04-24T17:27:01.655 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr ** Compaction Stats [default] ** 2024-04-24T17:27:01.655 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-24T17:27:01.655 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-24T17:27:01.655 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr 2024-04-24T17:27:01.655 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-24T17:27:01.655 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr 2024-04-24T17:27:01.655 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr Uptime(secs): 0.0 total, 0.0 interval 2024-04-24T17:27:01.655 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr Flush(GB): cumulative 0.000, interval 0.000 2024-04-24T17:27:01.655 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr AddFile(GB): cumulative 0.000, interval 0.000 2024-04-24T17:27:01.655 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr AddFile(Total Files): cumulative 0, interval 0 2024-04-24T17:27:01.655 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr AddFile(L0 Files): cumulative 0, interval 0 2024-04-24T17:27:01.655 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr AddFile(Keys): cumulative 0, interval 0 2024-04-24T17:27:01.655 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-24T17:27:01.655 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-24T17:27:01.655 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-24T17:27:01.656 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr Block cache BinnedLRUCache@0x555d0629d1f0#7 capacity: 512.00 MB usage: 0.00 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.8e-05 secs_since: 0 2024-04-24T17:27:01.656 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr Block cache entry stats(count,size,portion): Misc(1,0.00 KB,0%) 2024-04-24T17:27:01.656 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr 2024-04-24T17:27:01.656 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr ** File Read Latency Histogram By Level [default] ** 2024-04-24T17:27:01.656 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr 2024-04-24T17:27:01.656 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.867+0000 7f7d0ee60c80 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2024-04-24T17:27:01.656 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.867+0000 7f7d0ee60c80 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2024-04-24T17:27:01.656 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-24T17:27:00.867+0000 7f7d0ee60c80 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-a for mon.a 2024-04-24T17:27:01.656 INFO:teuthology.orchestra.run.smithi032.stdout:create mon.a on 2024-04-24T17:27:01.899 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-04-24T17:27:02.093 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /etc/systemd/system/ceph.target. 2024-04-24T17:27:02.322 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-a7981b84-025f-11ef-bc93-c7b262605968.target -> /etc/systemd/system/ceph-a7981b84-025f-11ef-bc93-c7b262605968.target. 2024-04-24T17:27:02.322 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-a7981b84-025f-11ef-bc93-c7b262605968.target -> /etc/systemd/system/ceph-a7981b84-025f-11ef-bc93-c7b262605968.target. 2024-04-24T17:27:02.595 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-a7981b84-025f-11ef-bc93-c7b262605968@mon.a 2024-04-24T17:27:02.595 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Failed to reset failed state of unit ceph-a7981b84-025f-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-a7981b84-025f-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-24T17:27:02.783 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-a7981b84-025f-11ef-bc93-c7b262605968.target.wants/ceph-a7981b84-025f-11ef-bc93-c7b262605968@mon.a.service -> /etc/systemd/system/ceph-a7981b84-025f-11ef-bc93-c7b262605968@.service. 2024-04-24T17:27:02.793 INFO:teuthology.orchestra.run.smithi032.stdout:firewalld does not appear to be present 2024-04-24T17:27:02.793 INFO:teuthology.orchestra.run.smithi032.stdout:Not possible to enable service . firewalld.service is not available 2024-04-24T17:27:02.793 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for mon to start... 2024-04-24T17:27:02.793 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for mon... 2024-04-24T17:27:04.250 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:04 smithi032 bash[20969]: cluster 2024-04-24T17:27:04.031415+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-04-24T17:27:04.940 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout cluster: 2024-04-24T17:27:04.940 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout id: a7981b84-025f-11ef-bc93-c7b262605968 2024-04-24T17:27:04.940 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-24T17:27:04.940 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-24T17:27:04.940 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout services: 2024-04-24T17:27:04.940 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.289178s) 2024-04-24T17:27:04.941 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-24T17:27:04.941 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-24T17:27:04.941 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-24T17:27:04.941 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout data: 2024-04-24T17:27:04.941 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-24T17:27:04.941 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-24T17:27:04.941 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-24T17:27:04.946 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout pgs: 2024-04-24T17:27:04.946 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-24T17:27:04.946 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.182+0000 7f2522495700 1 Processor -- start 2024-04-24T17:27:04.946 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.182+0000 7f2522495700 1 -- start start 2024-04-24T17:27:04.946 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.182+0000 7f2522495700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f251c10bcd0 con 0x7f251c1078a0 2024-04-24T17:27:04.946 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.182+0000 7f251bfff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f251c1078a0 0x7f251c107cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:45238/0 (socket says 172.21.15.32:45238) 2024-04-24T17:27:04.946 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.182+0000 7f251bfff700 1 -- 172.21.15.32:0/690784405 learned_addr learned my addr 172.21.15.32:0/690784405 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:27:04.946 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.182+0000 7f251affd700 1 -- 172.21.15.32:0/690784405 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 339159247 0 0) 0x7f251c10bcd0 con 0x7f251c1078a0 2024-04-24T17:27:04.946 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.182+0000 7f251affd700 1 -- 172.21.15.32:0/690784405 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25080035b0 con 0x7f251c1078a0 2024-04-24T17:27:04.947 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.182+0000 7f251affd700 1 -- 172.21.15.32:0/690784405 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2946630525 0 0) 0x7f25080035b0 con 0x7f251c1078a0 2024-04-24T17:27:04.947 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.182+0000 7f251affd700 1 -- 172.21.15.32:0/690784405 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f251c10ceb0 con 0x7f251c1078a0 2024-04-24T17:27:04.947 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.182+0000 7f251affd700 1 -- 172.21.15.32:0/690784405 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f250c001580 con 0x7f251c1078a0 2024-04-24T17:27:04.947 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.182+0000 7f251affd700 1 -- 172.21.15.32:0/690784405 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f250c0017e0 con 0x7f251c1078a0 2024-04-24T17:27:04.947 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.182+0000 7f251affd700 1 -- 172.21.15.32:0/690784405 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f250c001b60 con 0x7f251c1078a0 2024-04-24T17:27:04.947 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.182+0000 7f2522495700 1 -- 172.21.15.32:0/690784405 >> v1:172.21.15.32:6789/0 conn(0x7f251c1078a0 legacy=0x7f251c107cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:04.947 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.182+0000 7f2522495700 1 -- 172.21.15.32:0/690784405 shutdown_connections 2024-04-24T17:27:04.947 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.182+0000 7f2522495700 1 -- 172.21.15.32:0/690784405 >> 172.21.15.32:0/690784405 conn(0x7f251c102e40 msgr2=0x7f251c105280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:27:04.947 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.182+0000 7f2522495700 1 -- 172.21.15.32:0/690784405 shutdown_connections 2024-04-24T17:27:04.947 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.186+0000 7f2522495700 1 -- 172.21.15.32:0/690784405 wait complete. 2024-04-24T17:27:04.947 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.186+0000 7f2522495700 1 librados: starting msgr at 2024-04-24T17:27:04.947 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.186+0000 7f2522495700 1 librados: starting objecter 2024-04-24T17:27:04.947 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.186+0000 7f2522495700 1 Processor -- start 2024-04-24T17:27:04.947 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.186+0000 7f2522495700 1 -- start start 2024-04-24T17:27:04.947 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.186+0000 7f2522495700 1 librados: setting wanted keys 2024-04-24T17:27:04.948 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.186+0000 7f2522495700 1 librados: calling monclient init 2024-04-24T17:27:04.948 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.186+0000 7f2522495700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f251c19b340 con 0x7f251c1078a0 2024-04-24T17:27:04.948 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.186+0000 7f251bfff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f251c1078a0 0x7f251c19ac30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:45254/0 (socket says 172.21.15.32:45254) 2024-04-24T17:27:04.948 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.186+0000 7f251bfff700 1 -- 172.21.15.32:0/802509032 learned_addr learned my addr 172.21.15.32:0/802509032 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:27:04.948 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.186+0000 7f25197fa700 10 client.?.objecter ms_handle_connect 0x7f251c1078a0 2024-04-24T17:27:04.948 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.186+0000 7f25197fa700 10 client.?.objecter resend_mon_ops 2024-04-24T17:27:04.948 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.186+0000 7f25197fa700 1 -- 172.21.15.32:0/802509032 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 644185355 0 0) 0x7f251c19b340 con 0x7f251c1078a0 2024-04-24T17:27:04.948 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.186+0000 7f25197fa700 1 -- 172.21.15.32:0/802509032 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25080045b0 con 0x7f251c1078a0 2024-04-24T17:27:04.948 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.186+0000 7f25197fa700 1 -- 172.21.15.32:0/802509032 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 171238391 0 0) 0x7f25080045b0 con 0x7f251c1078a0 2024-04-24T17:27:04.948 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.186+0000 7f25197fa700 1 -- 172.21.15.32:0/802509032 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f251c19b340 con 0x7f251c1078a0 2024-04-24T17:27:04.948 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.186+0000 7f25197fa700 1 -- 172.21.15.32:0/802509032 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f250c001480 con 0x7f251c1078a0 2024-04-24T17:27:04.948 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.186+0000 7f25197fa700 1 -- 172.21.15.32:0/802509032 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2624852926 0 0) 0x7f251c19b340 con 0x7f251c1078a0 2024-04-24T17:27:04.948 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.186+0000 7f25197fa700 1 -- 172.21.15.32:0/802509032 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f251c19b510 con 0x7f251c1078a0 2024-04-24T17:27:04.948 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.186+0000 7f2522495700 1 -- 172.21.15.32:0/802509032 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f251c19b800 con 0x7f251c1078a0 2024-04-24T17:27:04.948 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.186+0000 7f2522495700 10 client.4098.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-24T17:27:04.948 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.186+0000 7f2522495700 1 -- 172.21.15.32:0/802509032 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f251c19f350 con 0x7f251c1078a0 2024-04-24T17:27:04.949 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.186+0000 7f2522495700 1 librados: init done 2024-04-24T17:27:04.949 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.186+0000 7f25197fa700 1 -- 172.21.15.32:0/802509032 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f250c0017e0 con 0x7f251c1078a0 2024-04-24T17:27:04.949 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.186+0000 7f25197fa700 1 -- 172.21.15.32:0/802509032 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f250c004c30 con 0x7f251c1078a0 2024-04-24T17:27:04.949 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.186+0000 7f25197fa700 1 -- 172.21.15.32:0/802509032 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f250c005160 con 0x7f251c1078a0 2024-04-24T17:27:04.949 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.186+0000 7f25197fa700 1 -- 172.21.15.32:0/802509032 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 2428325519 0 0) 0x7f250c0067a0 con 0x7f251c1078a0 2024-04-24T17:27:04.949 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.186+0000 7f25197fa700 10 client.4098.objecter ms_dispatch 0x7f251c001550 osd_map(1..1 src has 1..1) 2024-04-24T17:27:04.949 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.186+0000 7f25197fa700 3 client.4098.objecter handle_osd_map got epochs [1,1] > 0 2024-04-24T17:27:04.949 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.186+0000 7f25197fa700 3 client.4098.objecter handle_osd_map decoding full epoch 1 2024-04-24T17:27:04.949 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.186+0000 7f25197fa700 20 client.4098.objecter dump_active .. 0 homeless 2024-04-24T17:27:04.951 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.186+0000 7f2522495700 1 -- 172.21.15.32:0/802509032 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24fc005150 con 0x7f251c1078a0 2024-04-24T17:27:04.951 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.190+0000 7f25197fa700 1 -- 172.21.15.32:0/802509032 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (unknown 1092875540 0 2268998900) 0x7f250c006a40 con 0x7f251c1078a0 2024-04-24T17:27:04.951 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.322+0000 7f2522495700 1 -- 172.21.15.32:0/802509032 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "status"} v 0) -- 0x7f24fc005c50 con 0x7f251c1078a0 2024-04-24T17:27:04.951 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.322+0000 7f25197fa700 1 -- 172.21.15.32:0/802509032 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (unknown 1155462804 0 2973819976) 0x7f250c005690 con 0x7f251c1078a0 2024-04-24T17:27:04.951 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.322+0000 7f2522495700 10 librados: watch_flush enter 2024-04-24T17:27:04.951 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.322+0000 7f2522495700 10 librados: watch_flush exit 2024-04-24T17:27:04.951 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.322+0000 7f2522495700 20 client.4098.objecter shutdown clearing up homeless session... 2024-04-24T17:27:04.951 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.322+0000 7f2522495700 10 client.4098.objecter successfully canceled tick 2024-04-24T17:27:04.951 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.322+0000 7f2522495700 1 -- 172.21.15.32:0/802509032 >> v1:172.21.15.32:6789/0 conn(0x7f251c1078a0 legacy=0x7f251c19ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:04.951 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.322+0000 7f2522495700 1 -- 172.21.15.32:0/802509032 shutdown_connections 2024-04-24T17:27:04.951 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.322+0000 7f2522495700 1 -- 172.21.15.32:0/802509032 >> 172.21.15.32:0/802509032 conn(0x7f251c102e40 msgr2=0x7f251c191df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:27:04.951 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.322+0000 7f2522495700 1 -- 172.21.15.32:0/802509032 shutdown_connections 2024-04-24T17:27:04.951 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.322+0000 7f2522495700 1 -- 172.21.15.32:0/802509032 wait complete. 2024-04-24T17:27:04.952 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:04.322+0000 7f2522495700 1 librados: shutdown 2024-04-24T17:27:04.952 INFO:teuthology.orchestra.run.smithi032.stdout:mon is available 2024-04-24T17:27:04.952 INFO:teuthology.orchestra.run.smithi032.stdout:Assimilating anything we can from ceph.conf... 2024-04-24T17:27:05.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:05 smithi032 bash[20969]: cluster 2024-04-24T17:27:04.035773+0000 mon.a (mon.0) 2 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-04-24T17:27:05.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:05 smithi032 bash[20969]: cluster 2024-04-24T17:27:04.035773+0000 mon.a (mon.0) 2 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-04-24T17:27:05.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:05 smithi032 bash[20969]: cluster 2024-04-24T17:27:04.035953+0000 mon.a (mon.0) 3 : cluster [DBG] monmap epoch 1 2024-04-24T17:27:05.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:05 smithi032 bash[20969]: cluster 2024-04-24T17:27:04.035953+0000 mon.a (mon.0) 3 : cluster [DBG] monmap epoch 1 2024-04-24T17:27:05.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:05 smithi032 bash[20969]: cluster 2024-04-24T17:27:04.035966+0000 mon.a (mon.0) 4 : cluster [DBG] fsid a7981b84-025f-11ef-bc93-c7b262605968 2024-04-24T17:27:05.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:05 smithi032 bash[20969]: cluster 2024-04-24T17:27:04.035966+0000 mon.a (mon.0) 4 : cluster [DBG] fsid a7981b84-025f-11ef-bc93-c7b262605968 2024-04-24T17:27:05.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:05 smithi032 bash[20969]: cluster 2024-04-24T17:27:04.035973+0000 mon.a (mon.0) 5 : cluster [DBG] last_changed 2024-04-24T17:26:59.605652+0000 2024-04-24T17:27:05.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:05 smithi032 bash[20969]: cluster 2024-04-24T17:27:04.035973+0000 mon.a (mon.0) 5 : cluster [DBG] last_changed 2024-04-24T17:26:59.605652+0000 2024-04-24T17:27:05.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:05 smithi032 bash[20969]: cluster 2024-04-24T17:27:04.035979+0000 mon.a (mon.0) 6 : cluster [DBG] created 2024-04-24T17:26:59.605652+0000 2024-04-24T17:27:05.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:05 smithi032 bash[20969]: cluster 2024-04-24T17:27:04.035979+0000 mon.a (mon.0) 6 : cluster [DBG] created 2024-04-24T17:26:59.605652+0000 2024-04-24T17:27:05.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:05 smithi032 bash[20969]: cluster 2024-04-24T17:27:04.035988+0000 mon.a (mon.0) 7 : cluster [DBG] min_mon_release 19 (squid) 2024-04-24T17:27:05.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:05 smithi032 bash[20969]: cluster 2024-04-24T17:27:04.035988+0000 mon.a (mon.0) 7 : cluster [DBG] min_mon_release 19 (squid) 2024-04-24T17:27:05.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:05 smithi032 bash[20969]: cluster 2024-04-24T17:27:04.035995+0000 mon.a (mon.0) 8 : cluster [DBG] election_strategy: 1 2024-04-24T17:27:05.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:05 smithi032 bash[20969]: cluster 2024-04-24T17:27:04.035995+0000 mon.a (mon.0) 8 : cluster [DBG] election_strategy: 1 2024-04-24T17:27:05.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:05 smithi032 bash[20969]: cluster 2024-04-24T17:27:04.036003+0000 mon.a (mon.0) 9 : cluster [DBG] 0: v1:172.21.15.32:6789/0 mon.a 2024-04-24T17:27:05.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:05 smithi032 bash[20969]: cluster 2024-04-24T17:27:04.036003+0000 mon.a (mon.0) 9 : cluster [DBG] 0: v1:172.21.15.32:6789/0 mon.a 2024-04-24T17:27:05.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:05 smithi032 bash[20969]: cluster 2024-04-24T17:27:04.038301+0000 mon.a (mon.0) 10 : cluster [DBG] fsmap 2024-04-24T17:27:05.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:05 smithi032 bash[20969]: cluster 2024-04-24T17:27:04.038301+0000 mon.a (mon.0) 10 : cluster [DBG] fsmap 2024-04-24T17:27:05.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:05 smithi032 bash[20969]: cluster 2024-04-24T17:27:04.040757+0000 mon.a (mon.0) 11 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-04-24T17:27:05.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:05 smithi032 bash[20969]: cluster 2024-04-24T17:27:04.040757+0000 mon.a (mon.0) 11 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-04-24T17:27:05.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:05 smithi032 bash[20969]: cluster 2024-04-24T17:27:04.041250+0000 mon.a (mon.0) 12 : cluster [DBG] mgrmap e1: no daemons active 2024-04-24T17:27:05.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:05 smithi032 bash[20969]: cluster 2024-04-24T17:27:04.041250+0000 mon.a (mon.0) 12 : cluster [DBG] mgrmap e1: no daemons active 2024-04-24T17:27:05.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:05 smithi032 bash[20969]: audit 2024-04-24T17:27:04.324901+0000 mon.a (mon.0) 13 : audit [DBG] from='client.? v1:172.21.15.32:0/802509032' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-24T17:27:05.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:05 smithi032 bash[20969]: audit 2024-04-24T17:27:04.324901+0000 mon.a (mon.0) 13 : audit [DBG] from='client.? v1:172.21.15.32:0/802509032' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-24T17:27:06.250 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:06 smithi032 bash[20969]: audit 2024-04-24T17:27:05.926741+0000 mon.a (mon.0) 14 : audit [INF] from='client.? v1:172.21.15.32:0/3526809328' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-24T17:27:06.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:06 smithi032 bash[20969]: audit 2024-04-24T17:27:05.926741+0000 mon.a (mon.0) 14 : audit [INF] from='client.? v1:172.21.15.32:0/3526809328' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-24T17:27:06.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:06 smithi032 bash[20969]: audit 2024-04-24T17:27:05.931240+0000 mon.a (mon.0) 15 : audit [INF] from='client.? v1:172.21.15.32:0/3526809328' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-24T17:27:06.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:06 smithi032 bash[20969]: audit 2024-04-24T17:27:05.931240+0000 mon.a (mon.0) 15 : audit [INF] from='client.? v1:172.21.15.32:0/3526809328' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-24T17:27:06.559 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-24T17:27:06.560 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout [global] 2024-04-24T17:27:06.560 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout fsid = a7981b84-025f-11ef-bc93-c7b262605968 2024-04-24T17:27:06.560 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-24T17:27:06.560 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_host = [v1:172.21.15.32:6789] 2024-04-24T17:27:06.560 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-24T17:27:06.560 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-24T17:27:06.560 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-24T17:27:06.560 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-24T17:27:06.560 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-24T17:27:06.560 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-24T17:27:06.560 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-24T17:27:06.560 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-24T17:27:06.560 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout [osd] 2024-04-24T17:27:06.561 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-24T17:27:06.561 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-24T17:27:06.561 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.786+0000 7fe6722a5700 1 Processor -- start 2024-04-24T17:27:06.561 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.786+0000 7fe6722a5700 1 -- start start 2024-04-24T17:27:06.561 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.790+0000 7fe6722a5700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe66c07bab0 con 0x7fe66c07ceb0 2024-04-24T17:27:06.561 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.790+0000 7fe66bfff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe66c07ceb0 0x7fe66c07b310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:45270/0 (socket says 172.21.15.32:45270) 2024-04-24T17:27:06.561 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.790+0000 7fe66bfff700 1 -- 172.21.15.32:0/3193177978 learned_addr learned my addr 172.21.15.32:0/3193177978 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:27:06.561 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.790+0000 7fe66affd700 1 -- 172.21.15.32:0/3193177978 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2055195097 0 0) 0x7fe66c07bab0 con 0x7fe66c07ceb0 2024-04-24T17:27:06.561 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.790+0000 7fe66affd700 1 -- 172.21.15.32:0/3193177978 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6580035b0 con 0x7fe66c07ceb0 2024-04-24T17:27:06.561 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.790+0000 7fe66affd700 1 -- 172.21.15.32:0/3193177978 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3416646713 0 0) 0x7fe6580035b0 con 0x7fe66c07ceb0 2024-04-24T17:27:06.561 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.790+0000 7fe66affd700 1 -- 172.21.15.32:0/3193177978 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe66c07bc80 con 0x7fe66c07ceb0 2024-04-24T17:27:06.561 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.790+0000 7fe66affd700 1 -- 172.21.15.32:0/3193177978 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fe664001580 con 0x7fe66c07ceb0 2024-04-24T17:27:06.561 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.790+0000 7fe66affd700 1 -- 172.21.15.32:0/3193177978 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe664001850 con 0x7fe66c07ceb0 2024-04-24T17:27:06.561 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.790+0000 7fe66affd700 1 -- 172.21.15.32:0/3193177978 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fe664001bd0 con 0x7fe66c07ceb0 2024-04-24T17:27:06.561 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.790+0000 7fe6722a5700 1 -- 172.21.15.32:0/3193177978 >> v1:172.21.15.32:6789/0 conn(0x7fe66c07ceb0 legacy=0x7fe66c07b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:06.562 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.790+0000 7fe6722a5700 1 -- 172.21.15.32:0/3193177978 shutdown_connections 2024-04-24T17:27:06.562 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.790+0000 7fe6722a5700 1 -- 172.21.15.32:0/3193177978 >> 172.21.15.32:0/3193177978 conn(0x7fe66c102e30 msgr2=0x7fe66c105270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:27:06.562 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.790+0000 7fe6722a5700 1 -- 172.21.15.32:0/3193177978 shutdown_connections 2024-04-24T17:27:06.562 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.790+0000 7fe6722a5700 1 -- 172.21.15.32:0/3193177978 wait complete. 2024-04-24T17:27:06.562 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.790+0000 7fe6722a5700 1 librados: starting msgr at 2024-04-24T17:27:06.562 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.790+0000 7fe6722a5700 1 librados: starting objecter 2024-04-24T17:27:06.562 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.790+0000 7fe6722a5700 1 Processor -- start 2024-04-24T17:27:06.562 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.790+0000 7fe6722a5700 1 -- start start 2024-04-24T17:27:06.562 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.790+0000 7fe6722a5700 1 librados: setting wanted keys 2024-04-24T17:27:06.562 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.790+0000 7fe6722a5700 1 librados: calling monclient init 2024-04-24T17:27:06.562 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.790+0000 7fe6722a5700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe66c19d590 con 0x7fe66c07ceb0 2024-04-24T17:27:06.562 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.790+0000 7fe66bfff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe66c07ceb0 0x7fe66c19ce80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:45286/0 (socket says 172.21.15.32:45286) 2024-04-24T17:27:06.563 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.790+0000 7fe66bfff700 1 -- 172.21.15.32:0/3526809328 learned_addr learned my addr 172.21.15.32:0/3526809328 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:27:06.563 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.790+0000 7fe6697fa700 10 client.?.objecter ms_handle_connect 0x7fe66c07ceb0 2024-04-24T17:27:06.563 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.790+0000 7fe6697fa700 10 client.?.objecter resend_mon_ops 2024-04-24T17:27:06.563 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.790+0000 7fe6697fa700 1 -- 172.21.15.32:0/3526809328 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2936608733 0 0) 0x7fe66c19d590 con 0x7fe66c07ceb0 2024-04-24T17:27:06.563 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.790+0000 7fe6697fa700 1 -- 172.21.15.32:0/3526809328 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6580045b0 con 0x7fe66c07ceb0 2024-04-24T17:27:06.563 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.790+0000 7fe6697fa700 1 -- 172.21.15.32:0/3526809328 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 609543738 0 0) 0x7fe6580045b0 con 0x7fe66c07ceb0 2024-04-24T17:27:06.563 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.790+0000 7fe6697fa700 1 -- 172.21.15.32:0/3526809328 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe66c19d590 con 0x7fe66c07ceb0 2024-04-24T17:27:06.563 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.790+0000 7fe6697fa700 1 -- 172.21.15.32:0/3526809328 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fe664001540 con 0x7fe66c07ceb0 2024-04-24T17:27:06.563 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.794+0000 7fe6697fa700 1 -- 172.21.15.32:0/3526809328 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 702315547 0 0) 0x7fe66c19d590 con 0x7fe66c07ceb0 2024-04-24T17:27:06.563 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.794+0000 7fe6697fa700 1 -- 172.21.15.32:0/3526809328 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe66c19d760 con 0x7fe66c07ceb0 2024-04-24T17:27:06.563 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.794+0000 7fe6722a5700 1 -- 172.21.15.32:0/3526809328 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe66c19da50 con 0x7fe66c07ceb0 2024-04-24T17:27:06.563 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.794+0000 7fe6722a5700 10 client.4100.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-24T17:27:06.563 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.794+0000 7fe6722a5700 1 -- 172.21.15.32:0/3526809328 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe66c1a15a0 con 0x7fe66c07ceb0 2024-04-24T17:27:06.563 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.794+0000 7fe6722a5700 1 librados: init done 2024-04-24T17:27:06.564 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.794+0000 7fe6697fa700 1 -- 172.21.15.32:0/3526809328 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe664001850 con 0x7fe66c07ceb0 2024-04-24T17:27:06.564 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.794+0000 7fe6697fa700 1 -- 172.21.15.32:0/3526809328 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fe664004c30 con 0x7fe66c07ceb0 2024-04-24T17:27:06.564 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.794+0000 7fe6697fa700 1 -- 172.21.15.32:0/3526809328 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe6640051a0 con 0x7fe66c07ceb0 2024-04-24T17:27:06.564 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.794+0000 7fe6697fa700 1 -- 172.21.15.32:0/3526809328 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 2428325519 0 0) 0x7fe664006a10 con 0x7fe66c07ceb0 2024-04-24T17:27:06.564 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.794+0000 7fe6697fa700 10 client.4100.objecter ms_dispatch 0x7fe66c001550 osd_map(1..1 src has 1..1) 2024-04-24T17:27:06.564 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.794+0000 7fe6697fa700 3 client.4100.objecter handle_osd_map got epochs [1,1] > 0 2024-04-24T17:27:06.564 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.794+0000 7fe6697fa700 3 client.4100.objecter handle_osd_map decoding full epoch 1 2024-04-24T17:27:06.564 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.794+0000 7fe6697fa700 20 client.4100.objecter dump_active .. 0 homeless 2024-04-24T17:27:06.564 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.794+0000 7fe6722a5700 1 -- 172.21.15.32:0/3526809328 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe64c005150 con 0x7fe66c07ceb0 2024-04-24T17:27:06.564 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.794+0000 7fe6697fa700 1 -- 172.21.15.32:0/3526809328 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (unknown 1092875540 0 2268998900) 0x7fe664005620 con 0x7fe66c07ceb0 2024-04-24T17:27:06.564 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.922+0000 7fe6722a5700 1 -- 172.21.15.32:0/3526809328 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fe64c002430 con 0x7fe66c07ceb0 2024-04-24T17:27:06.564 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.930+0000 7fe6697fa700 1 -- 172.21.15.32:0/3526809328 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+353 (unknown 1213389831 0 1340751241) 0x7fe664005880 con 0x7fe66c07ceb0 2024-04-24T17:27:06.565 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.930+0000 7fe6722a5700 10 librados: watch_flush enter 2024-04-24T17:27:06.565 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.930+0000 7fe6722a5700 10 librados: watch_flush exit 2024-04-24T17:27:06.565 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.930+0000 7fe6722a5700 20 client.4100.objecter shutdown clearing up homeless session... 2024-04-24T17:27:06.565 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.930+0000 7fe6722a5700 10 client.4100.objecter successfully canceled tick 2024-04-24T17:27:06.565 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.930+0000 7fe6697fa700 1 -- 172.21.15.32:0/3526809328 <== mon.0 v1:172.21.15.32:6789/0 11 ==== config(25 keys) ==== 1011+0+0 (unknown 2148541123 0 0) 0x7fe664005f40 con 0x7fe66c07ceb0 2024-04-24T17:27:06.565 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.930+0000 7fe6722a5700 1 -- 172.21.15.32:0/3526809328 >> v1:172.21.15.32:6789/0 conn(0x7fe66c07ceb0 legacy=0x7fe66c19ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:06.565 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.930+0000 7fe6722a5700 1 -- 172.21.15.32:0/3526809328 shutdown_connections 2024-04-24T17:27:06.565 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.930+0000 7fe6722a5700 1 -- 172.21.15.32:0/3526809328 >> 172.21.15.32:0/3526809328 conn(0x7fe66c102e30 msgr2=0x7fe66c10c760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:27:06.565 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.930+0000 7fe6722a5700 1 -- 172.21.15.32:0/3526809328 shutdown_connections 2024-04-24T17:27:06.565 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.930+0000 7fe6722a5700 1 -- 172.21.15.32:0/3526809328 wait complete. 2024-04-24T17:27:06.565 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:05.930+0000 7fe6722a5700 1 librados: shutdown 2024-04-24T17:27:06.565 INFO:teuthology.orchestra.run.smithi032.stdout:Generating new minimal ceph.conf... 2024-04-24T17:27:08.000 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:07 smithi032 bash[20969]: audit 2024-04-24T17:27:07.516086+0000 mon.a (mon.0) 16 : audit [DBG] from='client.? v1:172.21.15.32:0/3629192925' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:27:08.001 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:07 smithi032 bash[20969]: audit 2024-04-24T17:27:07.516086+0000 mon.a (mon.0) 16 : audit [DBG] from='client.? v1:172.21.15.32:0/3629192925' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:27:08.176 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.374+0000 7f67f6731700 1 Processor -- start 2024-04-24T17:27:08.176 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.374+0000 7f67f6731700 1 -- start start 2024-04-24T17:27:08.176 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.374+0000 7f67f6731700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f67f010ded0 con 0x7f67f0109aa0 2024-04-24T17:27:08.177 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.374+0000 7f67effff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f67f0109aa0 0x7f67f0109ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:47294/0 (socket says 172.21.15.32:47294) 2024-04-24T17:27:08.177 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.374+0000 7f67effff700 1 -- 172.21.15.32:0/3407293251 learned_addr learned my addr 172.21.15.32:0/3407293251 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:27:08.177 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.374+0000 7f67ef7fe700 1 -- 172.21.15.32:0/3407293251 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4236487407 0 0) 0x7f67f010ded0 con 0x7f67f0109aa0 2024-04-24T17:27:08.177 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.374+0000 7f67ef7fe700 1 -- 172.21.15.32:0/3407293251 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67dc0035b0 con 0x7f67f0109aa0 2024-04-24T17:27:08.177 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.374+0000 7f67ef7fe700 1 -- 172.21.15.32:0/3407293251 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 384366149 0 0) 0x7f67dc0035b0 con 0x7f67f0109aa0 2024-04-24T17:27:08.179 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.374+0000 7f67ef7fe700 1 -- 172.21.15.32:0/3407293251 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67f010f0b0 con 0x7f67f0109aa0 2024-04-24T17:27:08.179 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.374+0000 7f67ef7fe700 1 -- 172.21.15.32:0/3407293251 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f67e0001580 con 0x7f67f0109aa0 2024-04-24T17:27:08.179 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.374+0000 7f67ef7fe700 1 -- 172.21.15.32:0/3407293251 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(25 keys) ==== 1011+0+0 (unknown 2148541123 0 0) 0x7f67e0001d10 con 0x7f67f0109aa0 2024-04-24T17:27:08.179 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.374+0000 7f67ef7fe700 1 -- 172.21.15.32:0/3407293251 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f67e0005960 con 0x7f67f0109aa0 2024-04-24T17:27:08.179 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.378+0000 7f67f6731700 1 -- 172.21.15.32:0/3407293251 >> v1:172.21.15.32:6789/0 conn(0x7f67f0109aa0 legacy=0x7f67f0109ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:08.179 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.378+0000 7f67f6731700 1 -- 172.21.15.32:0/3407293251 shutdown_connections 2024-04-24T17:27:08.180 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.378+0000 7f67f6731700 1 -- 172.21.15.32:0/3407293251 >> 172.21.15.32:0/3407293251 conn(0x7f67f007ce20 msgr2=0x7f67f01074a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:27:08.180 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.378+0000 7f67f6731700 1 -- 172.21.15.32:0/3407293251 shutdown_connections 2024-04-24T17:27:08.180 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.378+0000 7f67f6731700 1 -- 172.21.15.32:0/3407293251 wait complete. 2024-04-24T17:27:08.180 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.378+0000 7f67f6731700 1 librados: starting msgr at 2024-04-24T17:27:08.180 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.378+0000 7f67f6731700 1 librados: starting objecter 2024-04-24T17:27:08.180 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.378+0000 7f67f6731700 1 Processor -- start 2024-04-24T17:27:08.180 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.378+0000 7f67f6731700 1 -- start start 2024-04-24T17:27:08.180 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.378+0000 7f67f6731700 1 librados: setting wanted keys 2024-04-24T17:27:08.180 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.378+0000 7f67f6731700 1 librados: calling monclient init 2024-04-24T17:27:08.180 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.378+0000 7f67f6731700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f67f019dcf0 con 0x7f67f0109aa0 2024-04-24T17:27:08.180 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.378+0000 7f67effff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f67f0109aa0 0x7f67f019d5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:47304/0 (socket says 172.21.15.32:47304) 2024-04-24T17:27:08.180 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.378+0000 7f67effff700 1 -- 172.21.15.32:0/3629192925 learned_addr learned my addr 172.21.15.32:0/3629192925 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:27:08.180 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.378+0000 7f67edffb700 10 client.?.objecter ms_handle_connect 0x7f67f0109aa0 2024-04-24T17:27:08.180 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.378+0000 7f67edffb700 10 client.?.objecter resend_mon_ops 2024-04-24T17:27:08.184 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.378+0000 7f67edffb700 1 -- 172.21.15.32:0/3629192925 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 962903748 0 0) 0x7f67f019dcf0 con 0x7f67f0109aa0 2024-04-24T17:27:08.184 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.378+0000 7f67edffb700 1 -- 172.21.15.32:0/3629192925 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67dc0056e0 con 0x7f67f0109aa0 2024-04-24T17:27:08.184 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.378+0000 7f67edffb700 1 -- 172.21.15.32:0/3629192925 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3957136873 0 0) 0x7f67f010f0b0 con 0x7f67f0109aa0 2024-04-24T17:27:08.184 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.378+0000 7f67edffb700 1 -- 172.21.15.32:0/3629192925 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f67f019dcf0 con 0x7f67f0109aa0 2024-04-24T17:27:08.184 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.378+0000 7f67edffb700 1 -- 172.21.15.32:0/3629192925 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f67e0004870 con 0x7f67f0109aa0 2024-04-24T17:27:08.184 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.378+0000 7f67edffb700 1 -- 172.21.15.32:0/3629192925 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1149437006 0 0) 0x7f67f019dcf0 con 0x7f67f0109aa0 2024-04-24T17:27:08.184 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.378+0000 7f67edffb700 1 -- 172.21.15.32:0/3629192925 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67f019dec0 con 0x7f67f0109aa0 2024-04-24T17:27:08.184 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.378+0000 7f67f6731700 1 -- 172.21.15.32:0/3629192925 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f67f019e1b0 con 0x7f67f0109aa0 2024-04-24T17:27:08.184 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.378+0000 7f67f6731700 10 client.4102.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-24T17:27:08.184 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.378+0000 7f67f6731700 1 -- 172.21.15.32:0/3629192925 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f67f01a1d00 con 0x7f67f0109aa0 2024-04-24T17:27:08.184 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.378+0000 7f67f6731700 1 librados: init done 2024-04-24T17:27:08.184 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.378+0000 7f67edffb700 1 -- 172.21.15.32:0/3629192925 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(25 keys) ==== 1011+0+0 (unknown 2148541123 0 0) 0x7f67e00055e0 con 0x7f67f0109aa0 2024-04-24T17:27:08.185 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.378+0000 7f67edffb700 1 -- 172.21.15.32:0/3629192925 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f67e0005ef0 con 0x7f67f0109aa0 2024-04-24T17:27:08.185 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.378+0000 7f67edffb700 1 -- 172.21.15.32:0/3629192925 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f67e0006500 con 0x7f67f0109aa0 2024-04-24T17:27:08.185 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.378+0000 7f67edffb700 1 -- 172.21.15.32:0/3629192925 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 2428325519 0 0) 0x7f67e0007ca0 con 0x7f67f0109aa0 2024-04-24T17:27:08.185 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.378+0000 7f67edffb700 10 client.4102.objecter ms_dispatch 0x7f67f0001550 osd_map(1..1 src has 1..1) 2024-04-24T17:27:08.185 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.378+0000 7f67edffb700 3 client.4102.objecter handle_osd_map got epochs [1,1] > 0 2024-04-24T17:27:08.185 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.378+0000 7f67edffb700 3 client.4102.objecter handle_osd_map decoding full epoch 1 2024-04-24T17:27:08.185 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.378+0000 7f67edffb700 20 client.4102.objecter dump_active .. 0 homeless 2024-04-24T17:27:08.185 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.378+0000 7f67f6731700 1 -- 172.21.15.32:0/3629192925 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67d0005150 con 0x7f67f0109aa0 2024-04-24T17:27:08.185 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.382+0000 7f67edffb700 1 -- 172.21.15.32:0/3629192925 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (unknown 1092875540 0 2268998900) 0x7f67dc0056e0 con 0x7f67f0109aa0 2024-04-24T17:27:08.185 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.514+0000 7f67f6731700 1 -- 172.21.15.32:0/3629192925 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f67d0005440 con 0x7f67f0109aa0 2024-04-24T17:27:08.185 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.514+0000 7f67edffb700 1 -- 172.21.15.32:0/3629192925 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+147 (unknown 1452402520 0 1487960311) 0x7f67e0001b60 con 0x7f67f0109aa0 2024-04-24T17:27:08.185 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.514+0000 7f67f6731700 10 librados: watch_flush enter 2024-04-24T17:27:08.185 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.514+0000 7f67f6731700 10 librados: watch_flush exit 2024-04-24T17:27:08.185 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.514+0000 7f67f6731700 20 client.4102.objecter shutdown clearing up homeless session... 2024-04-24T17:27:08.185 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.514+0000 7f67f6731700 10 client.4102.objecter successfully canceled tick 2024-04-24T17:27:08.186 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.514+0000 7f67f6731700 1 -- 172.21.15.32:0/3629192925 >> v1:172.21.15.32:6789/0 conn(0x7f67f0109aa0 legacy=0x7f67f019d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:08.186 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.514+0000 7f67f6731700 1 -- 172.21.15.32:0/3629192925 shutdown_connections 2024-04-24T17:27:08.186 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.514+0000 7f67f6731700 1 -- 172.21.15.32:0/3629192925 >> 172.21.15.32:0/3629192925 conn(0x7f67f007ce20 msgr2=0x7f67f01074a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:27:08.186 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.514+0000 7f67f6731700 1 -- 172.21.15.32:0/3629192925 shutdown_connections 2024-04-24T17:27:08.186 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.514+0000 7f67f6731700 1 -- 172.21.15.32:0/3629192925 wait complete. 2024-04-24T17:27:08.186 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:07.514+0000 7f67f6731700 1 librados: shutdown 2024-04-24T17:27:08.186 INFO:teuthology.orchestra.run.smithi032.stdout:Restarting the monitor... 2024-04-24T17:27:08.501 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:08 smithi032 systemd[1]: Stopping Ceph mon.a for a7981b84-025f-11ef-bc93-c7b262605968... 2024-04-24T17:27:08.501 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:08 smithi032 bash[20969]: debug 2024-04-24T17:27:08.302+0000 7f93f19d7700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-04-24T17:27:08.501 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:08 smithi032 bash[20969]: debug 2024-04-24T17:27:08.302+0000 7f93f19d7700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-24T17:27:08.968 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:08 smithi032 bash[21358]: ceph-a7981b84-025f-11ef-bc93-c7b262605968-mon-a 2024-04-24T17:27:08.985 INFO:teuthology.orchestra.run.smithi032.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-24T17:27:09.250 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:08 smithi032 bash[21419]: Error response from daemon: No such container: ceph-a7981b84-025f-11ef-bc93-c7b262605968-mon-a 2024-04-24T17:27:09.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:08 smithi032 systemd[1]: ceph-a7981b84-025f-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-24T17:27:09.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:08 smithi032 systemd[1]: Stopped Ceph mon.a for a7981b84-025f-11ef-bc93-c7b262605968. 2024-04-24T17:27:09.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:08 smithi032 systemd[1]: Started Ceph mon.a for a7981b84-025f-11ef-bc93-c7b262605968. 2024-04-24T17:27:10.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.106+0000 7f7ed51c9c80 0 set uid:gid to 167:167 (ceph:ceph) 2024-04-24T17:27:10.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.106+0000 7f7ed51c9c80 0 ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev), process ceph-mon, pid 7 2024-04-24T17:27:10.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.106+0000 7f7ed51c9c80 0 pidfile_write: ignore empty --pid-file 2024-04-24T17:27:10.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 0 load: jerasure load: lrc 2024-04-24T17:27:10.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: RocksDB version: 7.9.2 2024-04-24T17:27:10.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Git sha 0 2024-04-24T17:27:10.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Compile date 2024-04-24 13:56:12 2024-04-24T17:27:10.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: DB SUMMARY 2024-04-24T17:27:10.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: DB Session ID: KRFFO1MZ34KRHXZIX387 2024-04-24T17:27:10.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: CURRENT file: CURRENT 2024-04-24T17:27:10.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: IDENTITY file: IDENTITY 2024-04-24T17:27:10.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-24T17:27:10.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-24T17:27:10.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 94375 ; 2024-04-24T17:27:10.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.error_if_exists: 0 2024-04-24T17:27:10.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.create_if_missing: 0 2024-04-24T17:27:10.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.paranoid_checks: 1 2024-04-24T17:27:10.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.flush_verify_memtable_count: 1 2024-04-24T17:27:10.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-24T17:27:10.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-24T17:27:10.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.env: 0x556f8b68fcc0 2024-04-24T17:27:10.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.fs: PosixFileSystem 2024-04-24T17:27:10.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.info_log: 0x556f8d7a3e00 2024-04-24T17:27:10.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.max_file_opening_threads: 16 2024-04-24T17:27:10.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.statistics: (nil) 2024-04-24T17:27:10.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.use_fsync: 0 2024-04-24T17:27:10.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.max_log_file_size: 0 2024-04-24T17:27:10.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-24T17:27:10.254 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.log_file_time_to_roll: 0 2024-04-24T17:27:10.254 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.keep_log_file_num: 1000 2024-04-24T17:27:10.254 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.recycle_log_file_num: 0 2024-04-24T17:27:10.254 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.allow_fallocate: 1 2024-04-24T17:27:10.254 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.allow_mmap_reads: 0 2024-04-24T17:27:10.254 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.allow_mmap_writes: 0 2024-04-24T17:27:10.254 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.use_direct_reads: 0 2024-04-24T17:27:10.254 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-24T17:27:10.254 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.create_missing_column_families: 0 2024-04-24T17:27:10.254 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.db_log_dir: 2024-04-24T17:27:10.254 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.wal_dir: 2024-04-24T17:27:10.254 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.table_cache_numshardbits: 6 2024-04-24T17:27:10.254 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-04-24T17:27:10.254 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-04-24T17:27:10.255 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-24T17:27:10.255 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-24T17:27:10.255 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-04-24T17:27:10.255 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.advise_random_on_open: 1 2024-04-24T17:27:10.255 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.db_write_buffer_size: 0 2024-04-24T17:27:10.255 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.write_buffer_manager: 0x556f8d7750e0 2024-04-24T17:27:10.255 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-24T17:27:10.255 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-24T17:27:10.255 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.use_adaptive_mutex: 0 2024-04-24T17:27:10.255 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.rate_limiter: (nil) 2024-04-24T17:27:10.255 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-24T17:27:10.255 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.wal_recovery_mode: 2 2024-04-24T17:27:10.255 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.enable_thread_tracking: 0 2024-04-24T17:27:10.255 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.enable_pipelined_write: 0 2024-04-24T17:27:10.255 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.unordered_write: 0 2024-04-24T17:27:10.255 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-24T17:27:10.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-24T17:27:10.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-24T17:27:10.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-24T17:27:10.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.row_cache: None 2024-04-24T17:27:10.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.wal_filter: None 2024-04-24T17:27:10.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-24T17:27:10.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.allow_ingest_behind: 0 2024-04-24T17:27:10.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.two_write_queues: 0 2024-04-24T17:27:10.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.manual_wal_flush: 0 2024-04-24T17:27:10.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.wal_compression: 0 2024-04-24T17:27:10.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.atomic_flush: 0 2024-04-24T17:27:10.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-24T17:27:10.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.persist_stats_to_disk: 0 2024-04-24T17:27:10.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-04-24T17:27:10.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.log_readahead_size: 0 2024-04-24T17:27:10.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-24T17:27:10.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.best_efforts_recovery: 0 2024-04-24T17:27:10.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-24T17:27:10.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-24T17:27:10.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.allow_data_in_errors: 0 2024-04-24T17:27:10.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.db_host_id: __hostname__ 2024-04-24T17:27:10.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.enforce_single_del_contracts: true 2024-04-24T17:27:10.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.max_background_jobs: 2 2024-04-24T17:27:10.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.max_background_compactions: -1 2024-04-24T17:27:10.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.max_subcompactions: 1 2024-04-24T17:27:10.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-24T17:27:10.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-24T17:27:10.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.delayed_write_rate : 16777216 2024-04-24T17:27:10.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.max_total_wal_size: 0 2024-04-24T17:27:10.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-24T17:27:10.257 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.stats_dump_period_sec: 600 2024-04-24T17:27:10.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.stats_persist_period_sec: 600 2024-04-24T17:27:10.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-24T17:27:10.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.max_open_files: -1 2024-04-24T17:27:10.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.bytes_per_sync: 0 2024-04-24T17:27:10.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-04-24T17:27:10.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-04-24T17:27:10.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.compaction_readahead_size: 0 2024-04-24T17:27:10.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.max_background_flushes: -1 2024-04-24T17:27:10.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Compression algorithms supported: 2024-04-24T17:27:10.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-24T17:27:10.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: kZSTD supported: 0 2024-04-24T17:27:10.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: kXpressCompression supported: 0 2024-04-24T17:27:10.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: kLZ4HCCompression supported: 1 2024-04-24T17:27:10.258 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: kZlibCompression supported: 1 2024-04-24T17:27:10.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: kSnappyCompression supported: 1 2024-04-24T17:27:10.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: kLZ4Compression supported: 1 2024-04-24T17:27:10.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: kBZip2Compression supported: 0 2024-04-24T17:27:10.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-04-24T17:27:10.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: DMutex implementation: pthread_mutex_t 2024-04-24T17:27:10.259 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-24T17:27:10.260 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-24T17:27:10.260 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-24T17:27:10.260 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.merge_operator: 2024-04-24T17:27:10.260 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.compaction_filter: None 2024-04-24T17:27:10.260 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.compaction_filter_factory: None 2024-04-24T17:27:10.260 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.sst_partitioner_factory: None 2024-04-24T17:27:10.261 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-04-24T17:27:10.261 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.table_factory: BlockBasedTable 2024-04-24T17:27:10.261 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556f8d7a3fc0) 2024-04-24T17:27:10.261 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: cache_index_and_filter_blocks: 1 2024-04-24T17:27:10.261 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-24T17:27:10.261 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-24T17:27:10.261 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: pin_top_level_index_and_filter: 1 2024-04-24T17:27:10.261 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: index_type: 0 2024-04-24T17:27:10.261 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: data_block_index_type: 0 2024-04-24T17:27:10.261 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: index_shortening: 1 2024-04-24T17:27:10.261 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: data_block_hash_table_util_ratio: 0.750000 2024-04-24T17:27:10.261 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: checksum: 4 2024-04-24T17:27:10.261 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: no_block_cache: 0 2024-04-24T17:27:10.261 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: block_cache: 0x556f8d7a51f0 2024-04-24T17:27:10.262 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: block_cache_name: BinnedLRUCache 2024-04-24T17:27:10.262 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: block_cache_options: 2024-04-24T17:27:10.262 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: capacity : 536870912 2024-04-24T17:27:10.262 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: num_shard_bits : 4 2024-04-24T17:27:10.262 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: strict_capacity_limit : 0 2024-04-24T17:27:10.262 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: high_pri_pool_ratio: 0.000 2024-04-24T17:27:10.262 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: block_cache_compressed: (nil) 2024-04-24T17:27:10.262 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: persistent_cache: (nil) 2024-04-24T17:27:10.262 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: block_size: 4096 2024-04-24T17:27:10.262 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: block_size_deviation: 10 2024-04-24T17:27:10.262 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: block_restart_interval: 16 2024-04-24T17:27:10.262 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: index_block_restart_interval: 1 2024-04-24T17:27:10.262 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: metadata_block_size: 4096 2024-04-24T17:27:10.263 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: partition_filters: 0 2024-04-24T17:27:10.263 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: use_delta_encoding: 1 2024-04-24T17:27:10.263 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: filter_policy: bloomfilter 2024-04-24T17:27:10.263 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: whole_key_filtering: 1 2024-04-24T17:27:10.263 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: verify_compression: 0 2024-04-24T17:27:10.263 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: read_amp_bytes_per_bit: 0 2024-04-24T17:27:10.263 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: format_version: 5 2024-04-24T17:27:10.263 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: enable_index_compression: 1 2024-04-24T17:27:10.263 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: block_align: 0 2024-04-24T17:27:10.263 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: max_auto_readahead_size: 262144 2024-04-24T17:27:10.263 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: prepopulate_block_cache: 0 2024-04-24T17:27:10.263 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: initial_auto_readahead_size: 8192 2024-04-24T17:27:10.263 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: num_file_reads_for_auto_readahead: 2 2024-04-24T17:27:10.263 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.write_buffer_size: 33554432 2024-04-24T17:27:10.263 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.max_write_buffer_number: 2 2024-04-24T17:27:10.263 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.compression: NoCompression 2024-04-24T17:27:10.264 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.bottommost_compression: Disabled 2024-04-24T17:27:10.264 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.prefix_extractor: nullptr 2024-04-24T17:27:10.264 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-24T17:27:10.264 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.num_levels: 7 2024-04-24T17:27:10.264 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-24T17:27:10.264 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-24T17:27:10.264 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-24T17:27:10.264 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-24T17:27:10.264 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-24T17:27:10.264 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-24T17:27:10.264 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-24T17:27:10.264 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-24T17:27:10.264 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-24T17:27:10.264 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-24T17:27:10.264 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-24T17:27:10.264 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-24T17:27:10.265 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.compression_opts.window_bits: -14 2024-04-24T17:27:10.265 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.compression_opts.level: 32767 2024-04-24T17:27:10.265 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.compression_opts.strategy: 0 2024-04-24T17:27:10.265 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-24T17:27:10.265 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-24T17:27:10.265 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-24T17:27:10.265 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-24T17:27:10.265 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.compression_opts.enabled: false 2024-04-24T17:27:10.265 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-24T17:27:10.265 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-24T17:27:10.265 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-24T17:27:10.265 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-24T17:27:10.265 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.target_file_size_base: 67108864 2024-04-24T17:27:10.265 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.target_file_size_multiplier: 1 2024-04-24T17:27:10.265 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-24T17:27:10.265 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-24T17:27:10.266 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-24T17:27:10.266 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-24T17:27:10.266 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-24T17:27:10.266 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-24T17:27:10.266 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-24T17:27:10.266 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-24T17:27:10.266 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-24T17:27:10.266 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-24T17:27:10.266 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-24T17:27:10.266 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-24T17:27:10.266 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-24T17:27:10.266 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.arena_block_size: 1048576 2024-04-24T17:27:10.266 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-24T17:27:10.266 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-24T17:27:10.266 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.disable_auto_compactions: 0 2024-04-24T17:27:10.266 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-24T17:27:10.267 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-24T17:27:10.267 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-24T17:27:10.267 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-24T17:27:10.267 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-24T17:27:10.267 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-24T17:27:10.267 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-24T17:27:10.267 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-24T17:27:10.267 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-24T17:27:10.267 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-24T17:27:10.267 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-24T17:27:10.267 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.inplace_update_support: 0 2024-04-24T17:27:10.267 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-04-24T17:27:10.267 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-24T17:27:10.267 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-24T17:27:10.267 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.memtable_huge_page_size: 0 2024-04-24T17:27:10.268 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.bloom_locality: 0 2024-04-24T17:27:10.268 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.max_successive_merges: 0 2024-04-24T17:27:10.268 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-04-24T17:27:10.268 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.paranoid_file_checks: 0 2024-04-24T17:27:10.268 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.force_consistency_checks: 1 2024-04-24T17:27:10.268 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.report_bg_io_stats: 0 2024-04-24T17:27:10.268 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.ttl: 2592000 2024-04-24T17:27:10.268 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-04-24T17:27:10.268 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-24T17:27:10.268 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-24T17:27:10.268 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.enable_blob_files: false 2024-04-24T17:27:10.268 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.min_blob_size: 0 2024-04-24T17:27:10.268 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.blob_file_size: 268435456 2024-04-24T17:27:10.268 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.blob_compression_type: NoCompression 2024-04-24T17:27:10.268 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.enable_blob_garbage_collection: false 2024-04-24T17:27:10.268 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-24T17:27:10.269 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-24T17:27:10.269 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-24T17:27:10.269 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.blob_file_starting_level: 0 2024-04-24T17:27:10.269 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.110+0000 7f7ed51c9c80 4 rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-24T17:27:10.269 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.114+0000 7f7ed51c9c80 4 rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-24T17:27:10.269 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.114+0000 7f7ed51c9c80 4 rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-24T17:27:10.269 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.114+0000 7f7ed51c9c80 4 rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 550a0adb-8b63-4cbc-9c5c-892975aaeb49 2024-04-24T17:27:10.269 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.114+0000 7f7ed51c9c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1713979630115131, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-24T17:27:10.269 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.114+0000 7f7ed51c9c80 4 rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-24T17:27:10.269 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.114+0000 7f7ed51c9c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1713979630116548, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 90113, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 292, "table_properties": {"data_size": 88175, "index_size": 258, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 12180, "raw_average_key_size": 48, "raw_value_size": 81543, "raw_average_value_size": 324, "num_data_blocks": 11, "num_entries": 251, "num_filter_entries": 251, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713979630, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "550a0adb-8b63-4cbc-9c5c-892975aaeb49", "db_session_id": "KRFFO1MZ34KRHXZIX387", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-24T17:27:10.269 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.114+0000 7f7ed51c9c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1713979630116655, "job": 1, "event": "recovery_finished"} 2024-04-24T17:27:10.269 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.114+0000 7f7ed51c9c80 4 rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-24T17:27:10.269 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.114+0000 7f7ed51c9c80 4 rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-24T17:27:10.269 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.114+0000 7f7ed51c9c80 4 rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x556f8d8c6000 2024-04-24T17:27:10.269 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.114+0000 7f7ed51c9c80 4 rocksdb: DB pointer 0x556f8d8b2000 2024-04-24T17:27:10.270 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.114+0000 7f7ec3921700 4 rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-24T17:27:10.270 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.114+0000 7f7ec3921700 4 rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-24T17:27:10.270 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: ** DB Stats ** 2024-04-24T17:27:10.270 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: Uptime(secs): 0.0 total, 0.0 interval 2024-04-24T17:27:10.270 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-24T17:27:10.270 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-24T17:27:10.270 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-24T17:27:10.270 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-24T17:27:10.270 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-24T17:27:10.270 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-24T17:27:10.270 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: ** Compaction Stats [default] ** 2024-04-24T17:27:10.270 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-24T17:27:10.270 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-24T17:27:10.270 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: L0 2/0 89.83 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 100.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-24T17:27:10.270 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: Sum 2/0 89.83 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 100.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-24T17:27:10.271 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 100.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-24T17:27:10.271 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: ** Compaction Stats [default] ** 2024-04-24T17:27:10.271 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-24T17:27:10.271 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-24T17:27:10.271 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 100.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-24T17:27:10.271 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-24T17:27:10.271 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: Uptime(secs): 0.0 total, 0.0 interval 2024-04-24T17:27:10.271 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: Flush(GB): cumulative 0.000, interval 0.000 2024-04-24T17:27:10.271 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-24T17:27:10.271 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: AddFile(Total Files): cumulative 0, interval 0 2024-04-24T17:27:10.271 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: AddFile(L0 Files): cumulative 0, interval 0 2024-04-24T17:27:10.271 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: AddFile(Keys): cumulative 0, interval 0 2024-04-24T17:27:10.271 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: Cumulative compaction: 0.00 GB write, 20.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-24T17:27:10.271 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: Interval compaction: 0.00 GB write, 20.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-24T17:27:10.272 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-24T17:27:10.272 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: Block cache BinnedLRUCache@0x556f8d7a51f0#7 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-04-24T17:27:10.272 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-04-24T17:27:10.272 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: ** File Read Latency Histogram By Level [default] ** 2024-04-24T17:27:10.272 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.118+0000 7f7ed51c9c80 0 starting mon.a rank 0 at public addrs v1:172.21.15.32:6789/0 at bind addrs v1:172.21.15.32:6789/0 mon_data /var/lib/ceph/mon/ceph-a fsid a7981b84-025f-11ef-bc93-c7b262605968 2024-04-24T17:27:10.272 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.118+0000 7f7ed51c9c80 1 mon.a@-1(???) e1 preinit fsid a7981b84-025f-11ef-bc93-c7b262605968 2024-04-24T17:27:10.272 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.118+0000 7f7ed51c9c80 0 mon.a@-1(???).mds e1 new map 2024-04-24T17:27:10.272 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.118+0000 7f7ed51c9c80 0 mon.a@-1(???).mds e1 print_map 2024-04-24T17:27:10.272 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: e1 2024-04-24T17:27:10.272 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: enable_multiple, ever_enabled_multiple: 1,1 2024-04-24T17:27:10.272 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-24T17:27:10.272 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: legacy client fscid: -1 2024-04-24T17:27:10.272 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: 2024-04-24T17:27:10.272 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: No filesystems configured 2024-04-24T17:27:10.272 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.118+0000 7f7ed51c9c80 0 mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-24T17:27:10.273 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.118+0000 7f7ed51c9c80 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-24T17:27:10.273 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.118+0000 7f7ed51c9c80 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-24T17:27:10.273 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.118+0000 7f7ed51c9c80 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-24T17:27:10.273 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: debug 2024-04-24T17:27:10.118+0000 7f7ed51c9c80 1 mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-24T17:27:10.273 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: cluster 2024-04-24T17:27:10.124243+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-04-24T17:27:10.273 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: cluster 2024-04-24T17:27:10.124243+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-04-24T17:27:10.273 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: cluster 2024-04-24T17:27:10.124313+0000 mon.a (mon.0) 2 : cluster [DBG] monmap epoch 1 2024-04-24T17:27:10.273 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: cluster 2024-04-24T17:27:10.124313+0000 mon.a (mon.0) 2 : cluster [DBG] monmap epoch 1 2024-04-24T17:27:10.273 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: cluster 2024-04-24T17:27:10.124323+0000 mon.a (mon.0) 3 : cluster [DBG] fsid a7981b84-025f-11ef-bc93-c7b262605968 2024-04-24T17:27:10.273 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: cluster 2024-04-24T17:27:10.124323+0000 mon.a (mon.0) 3 : cluster [DBG] fsid a7981b84-025f-11ef-bc93-c7b262605968 2024-04-24T17:27:10.273 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: cluster 2024-04-24T17:27:10.124342+0000 mon.a (mon.0) 4 : cluster [DBG] last_changed 2024-04-24T17:26:59.605652+0000 2024-04-24T17:27:10.273 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: cluster 2024-04-24T17:27:10.124342+0000 mon.a (mon.0) 4 : cluster [DBG] last_changed 2024-04-24T17:26:59.605652+0000 2024-04-24T17:27:10.273 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: cluster 2024-04-24T17:27:10.124353+0000 mon.a (mon.0) 5 : cluster [DBG] created 2024-04-24T17:26:59.605652+0000 2024-04-24T17:27:10.273 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: cluster 2024-04-24T17:27:10.124353+0000 mon.a (mon.0) 5 : cluster [DBG] created 2024-04-24T17:26:59.605652+0000 2024-04-24T17:27:10.273 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: cluster 2024-04-24T17:27:10.124366+0000 mon.a (mon.0) 6 : cluster [DBG] min_mon_release 19 (squid) 2024-04-24T17:27:10.273 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: cluster 2024-04-24T17:27:10.124366+0000 mon.a (mon.0) 6 : cluster [DBG] min_mon_release 19 (squid) 2024-04-24T17:27:10.274 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: cluster 2024-04-24T17:27:10.124375+0000 mon.a (mon.0) 7 : cluster [DBG] election_strategy: 1 2024-04-24T17:27:10.274 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: cluster 2024-04-24T17:27:10.124375+0000 mon.a (mon.0) 7 : cluster [DBG] election_strategy: 1 2024-04-24T17:27:10.274 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: cluster 2024-04-24T17:27:10.124394+0000 mon.a (mon.0) 8 : cluster [DBG] 0: v1:172.21.15.32:6789/0 mon.a 2024-04-24T17:27:10.274 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: cluster 2024-04-24T17:27:10.124394+0000 mon.a (mon.0) 8 : cluster [DBG] 0: v1:172.21.15.32:6789/0 mon.a 2024-04-24T17:27:10.274 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: cluster 2024-04-24T17:27:10.124785+0000 mon.a (mon.0) 9 : cluster [DBG] fsmap 2024-04-24T17:27:10.274 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: cluster 2024-04-24T17:27:10.124785+0000 mon.a (mon.0) 9 : cluster [DBG] fsmap 2024-04-24T17:27:10.274 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: cluster 2024-04-24T17:27:10.124818+0000 mon.a (mon.0) 10 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-04-24T17:27:10.274 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: cluster 2024-04-24T17:27:10.124818+0000 mon.a (mon.0) 10 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-04-24T17:27:10.274 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: cluster 2024-04-24T17:27:10.125120+0000 mon.a (mon.0) 11 : cluster [DBG] mgrmap e1: no daemons active 2024-04-24T17:27:10.274 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:10 smithi032 bash[21457]: cluster 2024-04-24T17:27:10.125120+0000 mon.a (mon.0) 11 : cluster [DBG] mgrmap e1: no daemons active 2024-04-24T17:27:11.014 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.154+0000 7f506e3a8700 1 Processor -- start 2024-04-24T17:27:11.014 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.154+0000 7f506e3a8700 1 -- start start 2024-04-24T17:27:11.014 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.154+0000 7f506e3a8700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f506807ade0 con 0x7f5068106170 2024-04-24T17:27:11.014 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.154+0000 7f5067fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f5068106170 0x7f5068106590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:47314/0 (socket says 172.21.15.32:47314) 2024-04-24T17:27:11.014 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.154+0000 7f5067fff700 1 -- 172.21.15.32:0/1537340973 learned_addr learned my addr 172.21.15.32:0/1537340973 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:27:11.015 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.154+0000 7f5066ffd700 1 -- 172.21.15.32:0/1537340973 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4001129339 0 0) 0x7f506807ade0 con 0x7f5068106170 2024-04-24T17:27:11.015 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.154+0000 7f5066ffd700 1 -- 172.21.15.32:0/1537340973 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50500035b0 con 0x7f5068106170 2024-04-24T17:27:11.015 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.154+0000 7f5066ffd700 1 -- 172.21.15.32:0/1537340973 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3955004236 0 0) 0x7f50500035b0 con 0x7f5068106170 2024-04-24T17:27:11.015 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.154+0000 7f5066ffd700 1 -- 172.21.15.32:0/1537340973 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5068107cb0 con 0x7f5068106170 2024-04-24T17:27:11.017 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.154+0000 7f5066ffd700 1 -- 172.21.15.32:0/1537340973 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f505c001580 con 0x7f5068106170 2024-04-24T17:27:11.017 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.154+0000 7f5066ffd700 1 -- 172.21.15.32:0/1537340973 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(25 keys) ==== 1011+0+0 (unknown 2148541123 0 0) 0x7f505c001d10 con 0x7f5068106170 2024-04-24T17:27:11.017 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.154+0000 7f5066ffd700 1 -- 172.21.15.32:0/1537340973 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f505c005960 con 0x7f5068106170 2024-04-24T17:27:11.017 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.154+0000 7f506e3a8700 1 -- 172.21.15.32:0/1537340973 >> v1:172.21.15.32:6789/0 conn(0x7f5068106170 legacy=0x7f5068106590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:11.017 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.154+0000 7f506e3a8700 1 -- 172.21.15.32:0/1537340973 shutdown_connections 2024-04-24T17:27:11.017 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.154+0000 7f506e3a8700 1 -- 172.21.15.32:0/1537340973 >> 172.21.15.32:0/1537340973 conn(0x7f5068101cd0 msgr2=0x7f5068104150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:27:11.017 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.154+0000 7f506e3a8700 1 -- 172.21.15.32:0/1537340973 shutdown_connections 2024-04-24T17:27:11.018 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.154+0000 7f506e3a8700 1 -- 172.21.15.32:0/1537340973 wait complete. 2024-04-24T17:27:11.018 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.158+0000 7f506e3a8700 1 librados: starting msgr at 2024-04-24T17:27:11.018 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.158+0000 7f506e3a8700 1 librados: starting objecter 2024-04-24T17:27:11.018 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.158+0000 7f506e3a8700 1 Processor -- start 2024-04-24T17:27:11.018 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.158+0000 7f506e3a8700 1 -- start start 2024-04-24T17:27:11.018 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.158+0000 7f506e3a8700 1 librados: setting wanted keys 2024-04-24T17:27:11.018 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.158+0000 7f506e3a8700 1 librados: calling monclient init 2024-04-24T17:27:11.018 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.158+0000 7f506e3a8700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f506819de10 con 0x7f5068106170 2024-04-24T17:27:11.018 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.158+0000 7f5067fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f5068106170 0x7f506819d700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:47326/0 (socket says 172.21.15.32:47326) 2024-04-24T17:27:11.018 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.158+0000 7f5067fff700 1 -- 172.21.15.32:0/3685254252 learned_addr learned my addr 172.21.15.32:0/3685254252 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:27:11.018 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.158+0000 7f50657fa700 10 client.?.objecter ms_handle_connect 0x7f5068106170 2024-04-24T17:27:11.018 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.158+0000 7f50657fa700 10 client.?.objecter resend_mon_ops 2024-04-24T17:27:11.018 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.158+0000 7f50657fa700 1 -- 172.21.15.32:0/3685254252 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1724696416 0 0) 0x7f506819de10 con 0x7f5068106170 2024-04-24T17:27:11.018 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.158+0000 7f50657fa700 1 -- 172.21.15.32:0/3685254252 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50500056e0 con 0x7f5068106170 2024-04-24T17:27:11.019 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.158+0000 7f50657fa700 1 -- 172.21.15.32:0/3685254252 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3161160630 0 0) 0x7f5068107cb0 con 0x7f5068106170 2024-04-24T17:27:11.022 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.158+0000 7f50657fa700 1 -- 172.21.15.32:0/3685254252 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f506819de10 con 0x7f5068106170 2024-04-24T17:27:11.022 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.158+0000 7f50657fa700 1 -- 172.21.15.32:0/3685254252 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f505c004870 con 0x7f5068106170 2024-04-24T17:27:11.022 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.158+0000 7f50657fa700 1 -- 172.21.15.32:0/3685254252 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 219708950 0 0) 0x7f506819de10 con 0x7f5068106170 2024-04-24T17:27:11.022 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.158+0000 7f50657fa700 1 -- 172.21.15.32:0/3685254252 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f506819dfe0 con 0x7f5068106170 2024-04-24T17:27:11.022 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.158+0000 7f506e3a8700 1 -- 172.21.15.32:0/3685254252 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f506819e2d0 con 0x7f5068106170 2024-04-24T17:27:11.022 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.158+0000 7f506e3a8700 10 client.14098.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-24T17:27:11.022 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.158+0000 7f506e3a8700 1 -- 172.21.15.32:0/3685254252 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f50681a1e20 con 0x7f5068106170 2024-04-24T17:27:11.022 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.158+0000 7f506e3a8700 1 librados: init done 2024-04-24T17:27:11.022 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.158+0000 7f50657fa700 1 -- 172.21.15.32:0/3685254252 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(25 keys) ==== 1011+0+0 (unknown 2148541123 0 0) 0x7f505c001b80 con 0x7f5068106170 2024-04-24T17:27:11.022 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.158+0000 7f50657fa700 1 -- 172.21.15.32:0/3685254252 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f505c005f10 con 0x7f5068106170 2024-04-24T17:27:11.023 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.158+0000 7f50657fa700 1 -- 172.21.15.32:0/3685254252 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f505c0064c0 con 0x7f5068106170 2024-04-24T17:27:11.023 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.158+0000 7f50657fa700 1 -- 172.21.15.32:0/3685254252 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 2428325519 0 0) 0x7f505c007bd0 con 0x7f5068106170 2024-04-24T17:27:11.023 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.158+0000 7f50657fa700 10 client.14098.objecter ms_dispatch 0x7f5068001550 osd_map(1..1 src has 1..1) 2024-04-24T17:27:11.023 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.158+0000 7f50657fa700 3 client.14098.objecter handle_osd_map got epochs [1,1] > 0 2024-04-24T17:27:11.023 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.158+0000 7f50657fa700 3 client.14098.objecter handle_osd_map decoding full epoch 1 2024-04-24T17:27:11.023 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.158+0000 7f50657fa700 20 client.14098.objecter dump_active .. 0 homeless 2024-04-24T17:27:11.023 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.158+0000 7f506e3a8700 1 -- 172.21.15.32:0/3685254252 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5054005150 con 0x7f5068106170 2024-04-24T17:27:11.023 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.162+0000 7f50657fa700 1 -- 172.21.15.32:0/3685254252 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (unknown 1092875540 0 2268998900) 0x7f50500056e0 con 0x7f5068106170 2024-04-24T17:27:11.023 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.294+0000 7f506e3a8700 1 -- 172.21.15.32:0/3685254252 --> v1:172.21.15.32:6789/0 -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f5054005440 con 0x7f5068106170 2024-04-24T17:27:11.023 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.298+0000 7f50657fa700 1 -- 172.21.15.32:0/3685254252 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (unknown 534127980 0 0) 0x7f505c005650 con 0x7f5068106170 2024-04-24T17:27:11.023 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.298+0000 7f506e3a8700 10 librados: watch_flush enter 2024-04-24T17:27:11.023 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.298+0000 7f506e3a8700 10 librados: watch_flush exit 2024-04-24T17:27:11.023 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.298+0000 7f506e3a8700 20 client.14098.objecter shutdown clearing up homeless session... 2024-04-24T17:27:11.023 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.298+0000 7f506e3a8700 10 client.14098.objecter successfully canceled tick 2024-04-24T17:27:11.024 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.298+0000 7f506e3a8700 1 -- 172.21.15.32:0/3685254252 >> v1:172.21.15.32:6789/0 conn(0x7f5068106170 legacy=0x7f506819d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:11.024 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.298+0000 7f506e3a8700 1 -- 172.21.15.32:0/3685254252 shutdown_connections 2024-04-24T17:27:11.024 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.298+0000 7f506e3a8700 1 -- 172.21.15.32:0/3685254252 >> 172.21.15.32:0/3685254252 conn(0x7f5068101cd0 msgr2=0x7f5068194320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:27:11.024 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.298+0000 7f506e3a8700 1 -- 172.21.15.32:0/3685254252 shutdown_connections 2024-04-24T17:27:11.024 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.298+0000 7f506e3a8700 1 -- 172.21.15.32:0/3685254252 wait complete. 2024-04-24T17:27:11.024 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:10.298+0000 7f506e3a8700 1 librados: shutdown 2024-04-24T17:27:11.025 INFO:teuthology.orchestra.run.smithi032.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-24T17:27:11.056 INFO:teuthology.orchestra.run.smithi032.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-24T17:27:11.056 INFO:teuthology.orchestra.run.smithi032.stdout:Creating mgr... 2024-04-24T17:27:11.057 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-24T17:27:11.057 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-24T17:27:11.390 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-a7981b84-025f-11ef-bc93-c7b262605968@mgr.y 2024-04-24T17:27:11.390 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Failed to reset failed state of unit ceph-a7981b84-025f-11ef-bc93-c7b262605968@mgr.y.service: Unit ceph-a7981b84-025f-11ef-bc93-c7b262605968@mgr.y.service not loaded. 2024-04-24T17:27:11.394 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:11 smithi032 bash[21457]: audit 2024-04-24T17:27:10.299261+0000 mon.a (mon.0) 12 : audit [INF] from='client.? v1:172.21.15.32:0/3685254252' entity='client.admin' 2024-04-24T17:27:11.394 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:11 smithi032 bash[21457]: audit 2024-04-24T17:27:10.299261+0000 mon.a (mon.0) 12 : audit [INF] from='client.? v1:172.21.15.32:0/3685254252' entity='client.admin' 2024-04-24T17:27:11.394 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:11 smithi032 systemd[1]: /etc/systemd/system/ceph-a7981b84-025f-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-24T17:27:11.395 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:11 smithi032 systemd[1]: /etc/systemd/system/ceph-a7981b84-025f-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-24T17:27:11.606 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-a7981b84-025f-11ef-bc93-c7b262605968.target.wants/ceph-a7981b84-025f-11ef-bc93-c7b262605968@mgr.y.service -> /etc/systemd/system/ceph-a7981b84-025f-11ef-bc93-c7b262605968@.service. 2024-04-24T17:27:11.615 INFO:teuthology.orchestra.run.smithi032.stdout:firewalld does not appear to be present 2024-04-24T17:27:11.615 INFO:teuthology.orchestra.run.smithi032.stdout:Not possible to enable service . firewalld.service is not available 2024-04-24T17:27:11.615 INFO:teuthology.orchestra.run.smithi032.stdout:firewalld does not appear to be present 2024-04-24T17:27:11.615 INFO:teuthology.orchestra.run.smithi032.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-24T17:27:11.615 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for mgr to start... 2024-04-24T17:27:11.615 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for mgr... 2024-04-24T17:27:11.750 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:11 smithi032 systemd[1]: /etc/systemd/system/ceph-a7981b84-025f-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-24T17:27:11.751 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:11 smithi032 systemd[1]: /etc/systemd/system/ceph-a7981b84-025f-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-24T17:27:11.751 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:11 smithi032 systemd[1]: Started Ceph mgr.y for a7981b84-025f-11ef-bc93-c7b262605968. 2024-04-24T17:27:13.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:13 smithi032 bash[21457]: audit 2024-04-24T17:27:13.116585+0000 mon.a (mon.0) 13 : audit [DBG] from='client.? v1:172.21.15.32:0/2372086827' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-24T17:27:13.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:13 smithi032 bash[21457]: audit 2024-04-24T17:27:13.116585+0000 mon.a (mon.0) 13 : audit [DBG] from='client.? v1:172.21.15.32:0/2372086827' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-24T17:27:13.251 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:12 smithi032 bash[21712]: debug 2024-04-24T17:27:12.954+0000 7f91b82a8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T17:27:13.776 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-24T17:27:13.776 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-04-24T17:27:13.776 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsid": "a7981b84-025f-11ef-bc93-c7b262605968", 2024-04-24T17:27:13.776 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "health": { 2024-04-24T17:27:13.776 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-24T17:27:13.776 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-24T17:27:13.776 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-24T17:27:13.776 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-24T17:27:13.776 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-24T17:27:13.777 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-24T17:27:13.777 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 0 2024-04-24T17:27:13.777 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-24T17:27:13.777 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-24T17:27:13.777 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "a" 2024-04-24T17:27:13.777 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-24T17:27:13.777 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-24T17:27:13.777 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-24T17:27:13.777 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T17:27:13.777 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-24T17:27:13.777 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-24T17:27:13.777 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-24T17:27:13.777 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-24T17:27:13.777 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T17:27:13.778 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-24T17:27:13.778 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-24T17:27:13.778 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-24T17:27:13.789 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-24T17:27:13.789 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-24T17:27:13.789 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-24T17:27:13.789 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-24T17:27:13.789 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-24T17:27:13.789 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-24T17:27:13.789 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-24T17:27:13.789 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-24T17:27:13.789 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-24T17:27:13.790 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-24T17:27:13.790 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-24T17:27:13.790 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-24T17:27:13.790 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-24T17:27:13.790 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-24T17:27:13.790 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-24T17:27:13.790 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T17:27:13.790 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-24T17:27:13.790 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-24T17:27:13.790 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-24T17:27:13.790 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-24T17:27:13.790 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-24T17:27:13.791 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-24T17:27:13.791 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-24T17:27:13.791 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-24T17:27:13.791 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-24T17:27:13.791 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "restful" 2024-04-24T17:27:13.791 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-24T17:27:13.791 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-24T17:27:13.791 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-24T17:27:13.791 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-24T17:27:13.791 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T17:27:13.791 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modified": "2024-04-24T17:27:04.037165+0000", 2024-04-24T17:27:13.791 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-24T17:27:13.791 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-24T17:27:13.792 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-24T17:27:13.792 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-04-24T17:27:13.792 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.970+0000 7f977b75c700 1 Processor -- start 2024-04-24T17:27:13.792 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.970+0000 7f977b75c700 1 -- start start 2024-04-24T17:27:13.792 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.970+0000 7f977b75c700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f977407ba50 con 0x7f977407ce50 2024-04-24T17:27:13.792 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.970+0000 7f97794f8700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f977407ce50 0x7f977407b2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:47362/0 (socket says 172.21.15.32:47362) 2024-04-24T17:27:13.792 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.970+0000 7f97794f8700 1 -- 172.21.15.32:0/3036093965 learned_addr learned my addr 172.21.15.32:0/3036093965 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:27:13.792 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.970+0000 7f9767fff700 1 -- 172.21.15.32:0/3036093965 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 660799219 0 0) 0x7f977407ba50 con 0x7f977407ce50 2024-04-24T17:27:13.792 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.970+0000 7f9767fff700 1 -- 172.21.15.32:0/3036093965 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97600035b0 con 0x7f977407ce50 2024-04-24T17:27:13.792 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.974+0000 7f9767fff700 1 -- 172.21.15.32:0/3036093965 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3932099076 0 0) 0x7f97600035b0 con 0x7f977407ce50 2024-04-24T17:27:13.792 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.974+0000 7f9767fff700 1 -- 172.21.15.32:0/3036093965 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f977407bc20 con 0x7f977407ce50 2024-04-24T17:27:13.792 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.974+0000 7f9767fff700 1 -- 172.21.15.32:0/3036093965 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f9768001580 con 0x7f977407ce50 2024-04-24T17:27:13.792 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.974+0000 7f9767fff700 1 -- 172.21.15.32:0/3036093965 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(25 keys) ==== 1011+0+0 (unknown 2148541123 0 0) 0x7f9768001c40 con 0x7f977407ce50 2024-04-24T17:27:13.792 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.974+0000 7f9767fff700 1 -- 172.21.15.32:0/3036093965 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f9768005960 con 0x7f977407ce50 2024-04-24T17:27:13.793 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.974+0000 7f977b75c700 1 -- 172.21.15.32:0/3036093965 >> v1:172.21.15.32:6789/0 conn(0x7f977407ce50 legacy=0x7f977407b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:13.793 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.974+0000 7f977b75c700 1 -- 172.21.15.32:0/3036093965 shutdown_connections 2024-04-24T17:27:13.793 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.974+0000 7f977b75c700 1 -- 172.21.15.32:0/3036093965 >> 172.21.15.32:0/3036093965 conn(0x7f9774102df0 msgr2=0x7f9774105230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:27:13.793 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.974+0000 7f977b75c700 1 -- 172.21.15.32:0/3036093965 shutdown_connections 2024-04-24T17:27:13.793 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.974+0000 7f977b75c700 1 -- 172.21.15.32:0/3036093965 wait complete. 2024-04-24T17:27:13.793 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.974+0000 7f977b75c700 1 librados: starting msgr at 2024-04-24T17:27:13.793 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.974+0000 7f977b75c700 1 librados: starting objecter 2024-04-24T17:27:13.793 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.974+0000 7f977b75c700 1 Processor -- start 2024-04-24T17:27:13.793 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.974+0000 7f977b75c700 1 -- start start 2024-04-24T17:27:13.793 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.974+0000 7f977b75c700 1 librados: setting wanted keys 2024-04-24T17:27:13.793 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.974+0000 7f977b75c700 1 librados: calling monclient init 2024-04-24T17:27:13.793 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.974+0000 7f977b75c700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f977419dd00 con 0x7f977407ce50 2024-04-24T17:27:13.793 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.974+0000 7f97794f8700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f977407ce50 0x7f977419d5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:47372/0 (socket says 172.21.15.32:47372) 2024-04-24T17:27:13.793 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.974+0000 7f97794f8700 1 -- 172.21.15.32:0/2372086827 learned_addr learned my addr 172.21.15.32:0/2372086827 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:27:13.793 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.974+0000 7f97667fc700 10 client.?.objecter ms_handle_connect 0x7f977407ce50 2024-04-24T17:27:13.793 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.974+0000 7f97667fc700 10 client.?.objecter resend_mon_ops 2024-04-24T17:27:13.794 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.974+0000 7f97667fc700 1 -- 172.21.15.32:0/2372086827 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 108152606 0 0) 0x7f977419dd00 con 0x7f977407ce50 2024-04-24T17:27:13.794 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.974+0000 7f97667fc700 1 -- 172.21.15.32:0/2372086827 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97600035b0 con 0x7f977407ce50 2024-04-24T17:27:13.794 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.974+0000 7f97667fc700 1 -- 172.21.15.32:0/2372086827 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 816381877 0 0) 0x7f97600035b0 con 0x7f977407ce50 2024-04-24T17:27:13.794 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.974+0000 7f97667fc700 1 -- 172.21.15.32:0/2372086827 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f977419dd00 con 0x7f977407ce50 2024-04-24T17:27:13.794 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.974+0000 7f97667fc700 1 -- 172.21.15.32:0/2372086827 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f9768001080 con 0x7f977407ce50 2024-04-24T17:27:13.794 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.974+0000 7f97667fc700 1 -- 172.21.15.32:0/2372086827 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3183836678 0 0) 0x7f977419dd00 con 0x7f977407ce50 2024-04-24T17:27:13.794 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.974+0000 7f97667fc700 1 -- 172.21.15.32:0/2372086827 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f977419ded0 con 0x7f977407ce50 2024-04-24T17:27:13.794 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.974+0000 7f977b75c700 1 -- 172.21.15.32:0/2372086827 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f977419e1c0 con 0x7f977407ce50 2024-04-24T17:27:13.794 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.974+0000 7f977b75c700 10 client.14102.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-24T17:27:13.794 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.974+0000 7f977b75c700 1 -- 172.21.15.32:0/2372086827 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f97741a1d10 con 0x7f977407ce50 2024-04-24T17:27:13.794 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.974+0000 7f97667fc700 1 -- 172.21.15.32:0/2372086827 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(25 keys) ==== 1011+0+0 (unknown 2148541123 0 0) 0x7f9768001a50 con 0x7f977407ce50 2024-04-24T17:27:13.794 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.974+0000 7f977b75c700 1 librados: init done 2024-04-24T17:27:13.794 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.978+0000 7f97667fc700 1 -- 172.21.15.32:0/2372086827 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f9768006270 con 0x7f977407ce50 2024-04-24T17:27:13.795 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.978+0000 7f97667fc700 1 -- 172.21.15.32:0/2372086827 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f97680068a0 con 0x7f977407ce50 2024-04-24T17:27:13.795 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.978+0000 7f97667fc700 1 -- 172.21.15.32:0/2372086827 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 2428325519 0 0) 0x7f9768007fe0 con 0x7f977407ce50 2024-04-24T17:27:13.795 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.978+0000 7f97667fc700 10 client.14102.objecter ms_dispatch 0x7f9774001550 osd_map(1..1 src has 1..1) 2024-04-24T17:27:13.795 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.978+0000 7f97667fc700 3 client.14102.objecter handle_osd_map got epochs [1,1] > 0 2024-04-24T17:27:13.795 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.978+0000 7f97667fc700 3 client.14102.objecter handle_osd_map decoding full epoch 1 2024-04-24T17:27:13.795 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.978+0000 7f97667fc700 20 client.14102.objecter dump_active .. 0 homeless 2024-04-24T17:27:13.795 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.978+0000 7f977b75c700 1 -- 172.21.15.32:0/2372086827 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f977410ee90 con 0x7f977407ce50 2024-04-24T17:27:13.795 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:12.978+0000 7f97667fc700 1 -- 172.21.15.32:0/2372086827 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (unknown 1092875540 0 2268998900) 0x7f9768001c00 con 0x7f977407ce50 2024-04-24T17:27:13.795 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:13.114+0000 7f977b75c700 1 -- 172.21.15.32:0/2372086827 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f97741a2210 con 0x7f977407ce50 2024-04-24T17:27:13.795 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:13.114+0000 7f97667fc700 1 -- 172.21.15.32:0/2372086827 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (unknown 4201413639 0 4205772144) 0x7f9768005590 con 0x7f977407ce50 2024-04-24T17:27:13.795 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:13.114+0000 7f977b75c700 10 librados: watch_flush enter 2024-04-24T17:27:13.795 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:13.114+0000 7f977b75c700 10 librados: watch_flush exit 2024-04-24T17:27:13.795 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:13.114+0000 7f977b75c700 20 client.14102.objecter shutdown clearing up homeless session... 2024-04-24T17:27:13.795 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:13.114+0000 7f977b75c700 10 client.14102.objecter successfully canceled tick 2024-04-24T17:27:13.795 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:13.114+0000 7f977b75c700 1 -- 172.21.15.32:0/2372086827 >> v1:172.21.15.32:6789/0 conn(0x7f977407ce50 legacy=0x7f977419d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:13.795 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:13.114+0000 7f977b75c700 1 -- 172.21.15.32:0/2372086827 shutdown_connections 2024-04-24T17:27:13.796 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:13.114+0000 7f977b75c700 1 -- 172.21.15.32:0/2372086827 >> 172.21.15.32:0/2372086827 conn(0x7f9774102df0 msgr2=0x7f977410bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:27:13.796 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:13.114+0000 7f977b75c700 1 -- 172.21.15.32:0/2372086827 shutdown_connections 2024-04-24T17:27:13.796 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:13.114+0000 7f977b75c700 1 -- 172.21.15.32:0/2372086827 wait complete. 2024-04-24T17:27:13.796 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:13.114+0000 7f977b75c700 1 librados: shutdown 2024-04-24T17:27:13.796 INFO:teuthology.orchestra.run.smithi032.stdout:mgr not available, waiting (1/15)... 2024-04-24T17:27:14.939 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:14 smithi032 bash[21712]: debug 2024-04-24T17:27:14.586+0000 7f91b82a8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T17:27:14.939 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:14 smithi032 bash[21712]: debug 2024-04-24T17:27:14.686+0000 7f91b82a8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T17:27:15.250 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:14 smithi032 bash[21712]: debug 2024-04-24T17:27:14.938+0000 7f91b82a8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T17:27:16.251 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:15 smithi032 bash[21712]: debug 2024-04-24T17:27:15.766+0000 7f91b82a8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T17:27:16.251 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:15 smithi032 bash[21712]: debug 2024-04-24T17:27:15.878+0000 7f91b82a8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T17:27:16.251 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:15 smithi032 bash[21712]: debug 2024-04-24T17:27:15.986+0000 7f91b82a8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T17:27:16.751 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:16 smithi032 bash[21712]: debug 2024-04-24T17:27:16.350+0000 7f91b82a8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-24T17:27:16.751 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:16 smithi032 bash[21712]: debug 2024-04-24T17:27:16.466+0000 7f91b82a8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T17:27:16.751 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:16 smithi032 bash[21712]: debug 2024-04-24T17:27:16.574+0000 7f91b82a8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T17:27:17.205 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:16 smithi032 bash[21712]: debug 2024-04-24T17:27:16.962+0000 7f91b82a8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T17:27:17.206 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:16 smithi032 bash[21457]: audit 2024-04-24T17:27:16.895085+0000 mon.a (mon.0) 14 : audit [DBG] from='client.? v1:172.21.15.32:0/2869269414' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-24T17:27:17.206 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:16 smithi032 bash[21457]: audit 2024-04-24T17:27:16.895085+0000 mon.a (mon.0) 14 : audit [DBG] from='client.? v1:172.21.15.32:0/2869269414' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-24T17:27:17.475 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:17 smithi032 bash[21712]: debug 2024-04-24T17:27:17.206+0000 7f91b82a8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T17:27:17.484 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-24T17:27:17.484 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-04-24T17:27:17.484 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsid": "a7981b84-025f-11ef-bc93-c7b262605968", 2024-04-24T17:27:17.484 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "health": { 2024-04-24T17:27:17.485 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-24T17:27:17.485 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-24T17:27:17.485 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-24T17:27:17.485 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-24T17:27:17.485 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-24T17:27:17.485 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-24T17:27:17.485 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 0 2024-04-24T17:27:17.485 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-24T17:27:17.485 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-24T17:27:17.485 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "a" 2024-04-24T17:27:17.485 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-24T17:27:17.485 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-04-24T17:27:17.485 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-24T17:27:17.485 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T17:27:17.486 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-24T17:27:17.486 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-24T17:27:17.486 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-24T17:27:17.486 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-24T17:27:17.486 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T17:27:17.486 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-24T17:27:17.486 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-24T17:27:17.486 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-24T17:27:17.486 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-24T17:27:17.486 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-24T17:27:17.486 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-24T17:27:17.486 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-24T17:27:17.486 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-24T17:27:17.487 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-24T17:27:17.487 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-24T17:27:17.487 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-24T17:27:17.487 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-24T17:27:17.487 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-24T17:27:17.487 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-24T17:27:17.487 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-24T17:27:17.487 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-24T17:27:17.487 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-24T17:27:17.487 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-24T17:27:17.487 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T17:27:17.487 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-24T17:27:17.487 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-24T17:27:17.487 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-24T17:27:17.487 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-24T17:27:17.488 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-24T17:27:17.488 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-24T17:27:17.488 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-24T17:27:17.488 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-24T17:27:17.488 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-24T17:27:17.488 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "restful" 2024-04-24T17:27:17.488 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-24T17:27:17.488 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-24T17:27:17.488 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-24T17:27:17.488 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-24T17:27:17.488 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T17:27:17.488 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modified": "2024-04-24T17:27:04.037165+0000", 2024-04-24T17:27:17.488 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-24T17:27:17.488 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-24T17:27:17.488 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-24T17:27:17.489 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-04-24T17:27:17.494 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.750+0000 7fc7fa3f3700 1 Processor -- start 2024-04-24T17:27:17.494 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.750+0000 7fc7fa3f3700 1 -- start start 2024-04-24T17:27:17.495 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.750+0000 7fc7fa3f3700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc7f407ba30 con 0x7fc7f407ce30 2024-04-24T17:27:17.495 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.750+0000 7fc7f3fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc7f407ce30 0x7fc7f407b290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34952/0 (socket says 172.21.15.32:34952) 2024-04-24T17:27:17.495 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.750+0000 7fc7f3fff700 1 -- 172.21.15.32:0/632070520 learned_addr learned my addr 172.21.15.32:0/632070520 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:27:17.495 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.750+0000 7fc7f2ffd700 1 -- 172.21.15.32:0/632070520 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2048713548 0 0) 0x7fc7f407ba30 con 0x7fc7f407ce30 2024-04-24T17:27:17.495 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.750+0000 7fc7f2ffd700 1 -- 172.21.15.32:0/632070520 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc7e00035b0 con 0x7fc7f407ce30 2024-04-24T17:27:17.495 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.750+0000 7fc7f2ffd700 1 -- 172.21.15.32:0/632070520 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 484861012 0 0) 0x7fc7e00035b0 con 0x7fc7f407ce30 2024-04-24T17:27:17.495 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.750+0000 7fc7f2ffd700 1 -- 172.21.15.32:0/632070520 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7f407bc00 con 0x7fc7f407ce30 2024-04-24T17:27:17.495 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.750+0000 7fc7f2ffd700 1 -- 172.21.15.32:0/632070520 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fc7e4001900 con 0x7fc7f407ce30 2024-04-24T17:27:17.495 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.754+0000 7fc7f2ffd700 1 -- 172.21.15.32:0/632070520 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(25 keys) ==== 1011+0+0 (unknown 2148541123 0 0) 0x7fc7e4001fc0 con 0x7fc7f407ce30 2024-04-24T17:27:17.495 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.754+0000 7fc7f2ffd700 1 -- 172.21.15.32:0/632070520 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fc7e4005ce0 con 0x7fc7f407ce30 2024-04-24T17:27:17.495 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.754+0000 7fc7fa3f3700 1 -- 172.21.15.32:0/632070520 >> v1:172.21.15.32:6789/0 conn(0x7fc7f407ce30 legacy=0x7fc7f407b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:17.495 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.754+0000 7fc7fa3f3700 1 -- 172.21.15.32:0/632070520 shutdown_connections 2024-04-24T17:27:17.496 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.754+0000 7fc7fa3f3700 1 -- 172.21.15.32:0/632070520 >> 172.21.15.32:0/632070520 conn(0x7fc7f4102e30 msgr2=0x7fc7f4105270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:27:17.496 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.754+0000 7fc7fa3f3700 1 -- 172.21.15.32:0/632070520 shutdown_connections 2024-04-24T17:27:17.496 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.754+0000 7fc7fa3f3700 1 -- 172.21.15.32:0/632070520 wait complete. 2024-04-24T17:27:17.496 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.754+0000 7fc7fa3f3700 1 librados: starting msgr at 2024-04-24T17:27:17.496 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.754+0000 7fc7fa3f3700 1 librados: starting objecter 2024-04-24T17:27:17.496 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.754+0000 7fc7fa3f3700 1 Processor -- start 2024-04-24T17:27:17.496 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.754+0000 7fc7fa3f3700 1 -- start start 2024-04-24T17:27:17.496 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.754+0000 7fc7fa3f3700 1 librados: setting wanted keys 2024-04-24T17:27:17.496 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.754+0000 7fc7fa3f3700 1 librados: calling monclient init 2024-04-24T17:27:17.496 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.754+0000 7fc7fa3f3700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc7f419dd30 con 0x7fc7f407ce30 2024-04-24T17:27:17.496 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.754+0000 7fc7f3fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc7f407ce30 0x7fc7f419d620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34964/0 (socket says 172.21.15.32:34964) 2024-04-24T17:27:17.496 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.754+0000 7fc7f3fff700 1 -- 172.21.15.32:0/2869269414 learned_addr learned my addr 172.21.15.32:0/2869269414 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:27:17.496 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.754+0000 7fc7f17fa700 10 client.?.objecter ms_handle_connect 0x7fc7f407ce30 2024-04-24T17:27:17.497 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.754+0000 7fc7f17fa700 10 client.?.objecter resend_mon_ops 2024-04-24T17:27:17.497 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.754+0000 7fc7f17fa700 1 -- 172.21.15.32:0/2869269414 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1357721388 0 0) 0x7fc7f419dd30 con 0x7fc7f407ce30 2024-04-24T17:27:17.497 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.754+0000 7fc7f17fa700 1 -- 172.21.15.32:0/2869269414 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc7e00035b0 con 0x7fc7f407ce30 2024-04-24T17:27:17.497 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.754+0000 7fc7f17fa700 1 -- 172.21.15.32:0/2869269414 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3820299505 0 0) 0x7fc7e00035b0 con 0x7fc7f407ce30 2024-04-24T17:27:17.497 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.754+0000 7fc7f17fa700 1 -- 172.21.15.32:0/2869269414 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc7f419dd30 con 0x7fc7f407ce30 2024-04-24T17:27:17.497 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.754+0000 7fc7f17fa700 1 -- 172.21.15.32:0/2869269414 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fc7e4001400 con 0x7fc7f407ce30 2024-04-24T17:27:17.497 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.754+0000 7fc7f17fa700 1 -- 172.21.15.32:0/2869269414 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3245605570 0 0) 0x7fc7f419dd30 con 0x7fc7f407ce30 2024-04-24T17:27:17.497 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.754+0000 7fc7f17fa700 1 -- 172.21.15.32:0/2869269414 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7f419df00 con 0x7fc7f407ce30 2024-04-24T17:27:17.497 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.754+0000 7fc7fa3f3700 1 -- 172.21.15.32:0/2869269414 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc7f419e1f0 con 0x7fc7f407ce30 2024-04-24T17:27:17.497 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.754+0000 7fc7fa3f3700 10 client.14104.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-24T17:27:17.497 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.754+0000 7fc7fa3f3700 1 -- 172.21.15.32:0/2869269414 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc7f41a1d40 con 0x7fc7f407ce30 2024-04-24T17:27:17.497 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.754+0000 7fc7fa3f3700 1 librados: init done 2024-04-24T17:27:17.497 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.754+0000 7fc7f17fa700 1 -- 172.21.15.32:0/2869269414 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(25 keys) ==== 1011+0+0 (unknown 2148541123 0 0) 0x7fc7e4005910 con 0x7fc7f407ce30 2024-04-24T17:27:17.497 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.754+0000 7fc7f17fa700 1 -- 172.21.15.32:0/2869269414 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fc7e4006400 con 0x7fc7f407ce30 2024-04-24T17:27:17.498 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.754+0000 7fc7f17fa700 1 -- 172.21.15.32:0/2869269414 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc7e4006a30 con 0x7fc7f407ce30 2024-04-24T17:27:17.498 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.754+0000 7fc7f17fa700 1 -- 172.21.15.32:0/2869269414 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 2428325519 0 0) 0x7fc7e4008170 con 0x7fc7f407ce30 2024-04-24T17:27:17.498 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.754+0000 7fc7f17fa700 10 client.14104.objecter ms_dispatch 0x7fc7f4001550 osd_map(1..1 src has 1..1) 2024-04-24T17:27:17.498 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.754+0000 7fc7f17fa700 3 client.14104.objecter handle_osd_map got epochs [1,1] > 0 2024-04-24T17:27:17.498 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.754+0000 7fc7f17fa700 3 client.14104.objecter handle_osd_map decoding full epoch 1 2024-04-24T17:27:17.498 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.754+0000 7fc7f17fa700 20 client.14104.objecter dump_active .. 0 homeless 2024-04-24T17:27:17.498 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.758+0000 7fc7fa3f3700 1 -- 172.21.15.32:0/2869269414 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7d4005150 con 0x7fc7f407ce30 2024-04-24T17:27:17.498 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.758+0000 7fc7f17fa700 1 -- 172.21.15.32:0/2869269414 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (unknown 1092875540 0 2268998900) 0x7fc7e4001dd0 con 0x7fc7f407ce30 2024-04-24T17:27:17.498 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.890+0000 7fc7fa3f3700 1 -- 172.21.15.32:0/2869269414 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fc7d4005440 con 0x7fc7f407ce30 2024-04-24T17:27:17.498 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.894+0000 7fc7f17fa700 1 -- 172.21.15.32:0/2869269414 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (unknown 4201413639 0 3303094643) 0x7fc7e4004d50 con 0x7fc7f407ce30 2024-04-24T17:27:17.498 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.894+0000 7fc7fa3f3700 10 librados: watch_flush enter 2024-04-24T17:27:17.498 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.894+0000 7fc7fa3f3700 10 librados: watch_flush exit 2024-04-24T17:27:17.498 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.894+0000 7fc7fa3f3700 20 client.14104.objecter shutdown clearing up homeless session... 2024-04-24T17:27:17.499 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.894+0000 7fc7fa3f3700 10 client.14104.objecter successfully canceled tick 2024-04-24T17:27:17.499 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.894+0000 7fc7fa3f3700 1 -- 172.21.15.32:0/2869269414 >> v1:172.21.15.32:6789/0 conn(0x7fc7f407ce30 legacy=0x7fc7f419d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:17.499 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.894+0000 7fc7fa3f3700 1 -- 172.21.15.32:0/2869269414 shutdown_connections 2024-04-24T17:27:17.499 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.894+0000 7fc7fa3f3700 1 -- 172.21.15.32:0/2869269414 >> 172.21.15.32:0/2869269414 conn(0x7fc7f4102e30 msgr2=0x7fc7f410bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:27:17.499 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.894+0000 7fc7fa3f3700 1 -- 172.21.15.32:0/2869269414 shutdown_connections 2024-04-24T17:27:17.499 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.894+0000 7fc7fa3f3700 1 -- 172.21.15.32:0/2869269414 wait complete. 2024-04-24T17:27:17.499 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:16.894+0000 7fc7fa3f3700 1 librados: shutdown 2024-04-24T17:27:17.499 INFO:teuthology.orchestra.run.smithi032.stdout:mgr not available, waiting (2/15)... 2024-04-24T17:27:17.751 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:17 smithi032 bash[21712]: debug 2024-04-24T17:27:17.474+0000 7f91b82a8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T17:27:17.751 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:17 smithi032 bash[21712]: debug 2024-04-24T17:27:17.590+0000 7f91b82a8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T17:27:18.251 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:17 smithi032 bash[21712]: debug 2024-04-24T17:27:17.870+0000 7f91b82a8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T17:27:18.661 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:18 smithi032 bash[21712]: debug 2024-04-24T17:27:18.293+0000 7f91b82a8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T17:27:18.661 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:18 smithi032 bash[21712]: debug 2024-04-24T17:27:18.537+0000 7f91b82a8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T17:27:18.914 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:18 smithi032 bash[21712]: debug 2024-04-24T17:27:18.661+0000 7f91b82a8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T17:27:19.250 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:18 smithi032 bash[21712]: debug 2024-04-24T17:27:18.913+0000 7f91b82a8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T17:27:20.216 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:19 smithi032 bash[21712]: debug 2024-04-24T17:27:19.957+0000 7f91b82a8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T17:27:20.216 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:20 smithi032 bash[21712]: debug 2024-04-24T17:27:20.077+0000 7f91b82a8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T17:27:20.501 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:20 smithi032 bash[21712]: debug 2024-04-24T17:27:20.385+0000 7f91b82a8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T17:27:20.777 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:20 smithi032 bash[21712]: debug 2024-04-24T17:27:20.505+0000 7f91b82a8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T17:27:20.778 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:20 smithi032 bash[21457]: audit 2024-04-24T17:27:20.553097+0000 mon.a (mon.0) 15 : audit [DBG] from='client.? v1:172.21.15.32:0/1866883012' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-24T17:27:20.778 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:20 smithi032 bash[21457]: audit 2024-04-24T17:27:20.553097+0000 mon.a (mon.0) 15 : audit [DBG] from='client.? v1:172.21.15.32:0/1866883012' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-24T17:27:21.104 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:20 smithi032 bash[21712]: debug 2024-04-24T17:27:20.981+0000 7f91b82a8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T17:27:21.216 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-24T17:27:21.216 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-04-24T17:27:21.216 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsid": "a7981b84-025f-11ef-bc93-c7b262605968", 2024-04-24T17:27:21.216 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "health": { 2024-04-24T17:27:21.217 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-24T17:27:21.217 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-24T17:27:21.217 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-24T17:27:21.217 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-24T17:27:21.217 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-24T17:27:21.217 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-24T17:27:21.217 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 0 2024-04-24T17:27:21.217 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-24T17:27:21.217 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-24T17:27:21.217 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "a" 2024-04-24T17:27:21.217 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-24T17:27:21.217 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-04-24T17:27:21.217 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-24T17:27:21.218 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T17:27:21.218 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-24T17:27:21.218 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-24T17:27:21.218 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-24T17:27:21.218 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-24T17:27:21.218 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T17:27:21.218 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-24T17:27:21.218 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-24T17:27:21.227 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-24T17:27:21.227 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-24T17:27:21.227 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-24T17:27:21.227 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-24T17:27:21.227 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-24T17:27:21.227 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-24T17:27:21.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-24T17:27:21.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-24T17:27:21.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-24T17:27:21.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-24T17:27:21.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-24T17:27:21.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-24T17:27:21.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-24T17:27:21.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-24T17:27:21.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-24T17:27:21.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-24T17:27:21.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T17:27:21.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-24T17:27:21.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-24T17:27:21.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-24T17:27:21.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-24T17:27:21.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-24T17:27:21.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-24T17:27:21.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-24T17:27:21.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-24T17:27:21.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-24T17:27:21.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "restful" 2024-04-24T17:27:21.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-24T17:27:21.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-24T17:27:21.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-24T17:27:21.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-24T17:27:21.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T17:27:21.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modified": "2024-04-24T17:27:04.037165+0000", 2024-04-24T17:27:21.230 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-24T17:27:21.230 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-24T17:27:21.230 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-24T17:27:21.230 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-04-24T17:27:21.230 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.405+0000 7fc9de1d1700 1 Processor -- start 2024-04-24T17:27:21.230 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.405+0000 7fc9de1d1700 1 -- start start 2024-04-24T17:27:21.230 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.405+0000 7fc9de1d1700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc9d807ae40 con 0x7fc9d8106110 2024-04-24T17:27:21.230 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.405+0000 7fc9d77fe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc9d8106110 0x7fc9d8106530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34978/0 (socket says 172.21.15.32:34978) 2024-04-24T17:27:21.230 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.405+0000 7fc9d77fe700 1 -- 172.21.15.32:0/204375263 learned_addr learned my addr 172.21.15.32:0/204375263 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:27:21.230 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.405+0000 7fc9d67fc700 1 -- 172.21.15.32:0/204375263 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3818769550 0 0) 0x7fc9d807ae40 con 0x7fc9d8106110 2024-04-24T17:27:21.230 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.405+0000 7fc9d67fc700 1 -- 172.21.15.32:0/204375263 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9c00035b0 con 0x7fc9d8106110 2024-04-24T17:27:21.230 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.405+0000 7fc9d67fc700 1 -- 172.21.15.32:0/204375263 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2964446488 0 0) 0x7fc9c00035b0 con 0x7fc9d8106110 2024-04-24T17:27:21.231 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.405+0000 7fc9d67fc700 1 -- 172.21.15.32:0/204375263 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9d8107c50 con 0x7fc9d8106110 2024-04-24T17:27:21.231 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.405+0000 7fc9d67fc700 1 -- 172.21.15.32:0/204375263 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fc9c8001900 con 0x7fc9d8106110 2024-04-24T17:27:21.231 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.405+0000 7fc9d67fc700 1 -- 172.21.15.32:0/204375263 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(25 keys) ==== 1011+0+0 (unknown 2148541123 0 0) 0x7fc9c8002090 con 0x7fc9d8106110 2024-04-24T17:27:21.231 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.405+0000 7fc9d67fc700 1 -- 172.21.15.32:0/204375263 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fc9c8005ce0 con 0x7fc9d8106110 2024-04-24T17:27:21.231 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.405+0000 7fc9de1d1700 1 -- 172.21.15.32:0/204375263 >> v1:172.21.15.32:6789/0 conn(0x7fc9d8106110 legacy=0x7fc9d8106530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:21.231 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.405+0000 7fc9de1d1700 1 -- 172.21.15.32:0/204375263 shutdown_connections 2024-04-24T17:27:21.231 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.405+0000 7fc9de1d1700 1 -- 172.21.15.32:0/204375263 >> 172.21.15.32:0/204375263 conn(0x7fc9d8101cd0 msgr2=0x7fc9d8104150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:27:21.231 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.405+0000 7fc9de1d1700 1 -- 172.21.15.32:0/204375263 shutdown_connections 2024-04-24T17:27:21.231 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.405+0000 7fc9de1d1700 1 -- 172.21.15.32:0/204375263 wait complete. 2024-04-24T17:27:21.231 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.409+0000 7fc9de1d1700 1 librados: starting msgr at 2024-04-24T17:27:21.231 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.409+0000 7fc9de1d1700 1 librados: starting objecter 2024-04-24T17:27:21.231 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.409+0000 7fc9de1d1700 1 Processor -- start 2024-04-24T17:27:21.231 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.409+0000 7fc9de1d1700 1 -- start start 2024-04-24T17:27:21.232 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.409+0000 7fc9de1d1700 1 librados: setting wanted keys 2024-04-24T17:27:21.232 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.409+0000 7fc9de1d1700 1 librados: calling monclient init 2024-04-24T17:27:21.232 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.409+0000 7fc9de1d1700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc9d819ddf0 con 0x7fc9d8106110 2024-04-24T17:27:21.232 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.409+0000 7fc9d77fe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc9d8106110 0x7fc9d819d6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34994/0 (socket says 172.21.15.32:34994) 2024-04-24T17:27:21.232 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.409+0000 7fc9d77fe700 1 -- 172.21.15.32:0/1866883012 learned_addr learned my addr 172.21.15.32:0/1866883012 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:27:21.232 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.409+0000 7fc9d4ff9700 10 client.?.objecter ms_handle_connect 0x7fc9d8106110 2024-04-24T17:27:21.232 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.409+0000 7fc9d4ff9700 10 client.?.objecter resend_mon_ops 2024-04-24T17:27:21.232 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.409+0000 7fc9d4ff9700 1 -- 172.21.15.32:0/1866883012 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3966822915 0 0) 0x7fc9d819ddf0 con 0x7fc9d8106110 2024-04-24T17:27:21.232 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.409+0000 7fc9d4ff9700 1 -- 172.21.15.32:0/1866883012 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9c00056e0 con 0x7fc9d8106110 2024-04-24T17:27:21.232 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.409+0000 7fc9d4ff9700 1 -- 172.21.15.32:0/1866883012 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3161246789 0 0) 0x7fc9d8107c50 con 0x7fc9d8106110 2024-04-24T17:27:21.232 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.409+0000 7fc9d4ff9700 1 -- 172.21.15.32:0/1866883012 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc9d819ddf0 con 0x7fc9d8106110 2024-04-24T17:27:21.232 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.409+0000 7fc9d4ff9700 1 -- 172.21.15.32:0/1866883012 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fc9c8004bf0 con 0x7fc9d8106110 2024-04-24T17:27:21.232 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.409+0000 7fc9d4ff9700 1 -- 172.21.15.32:0/1866883012 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 153345556 0 0) 0x7fc9d819ddf0 con 0x7fc9d8106110 2024-04-24T17:27:21.232 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.409+0000 7fc9d4ff9700 1 -- 172.21.15.32:0/1866883012 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9d819dfc0 con 0x7fc9d8106110 2024-04-24T17:27:21.233 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.409+0000 7fc9de1d1700 1 -- 172.21.15.32:0/1866883012 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc9d819e2b0 con 0x7fc9d8106110 2024-04-24T17:27:21.233 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.409+0000 7fc9de1d1700 10 client.14106.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-24T17:27:21.233 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.409+0000 7fc9de1d1700 1 -- 172.21.15.32:0/1866883012 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc9d81a1e00 con 0x7fc9d8106110 2024-04-24T17:27:21.233 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.409+0000 7fc9de1d1700 1 librados: init done 2024-04-24T17:27:21.233 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.409+0000 7fc9d4ff9700 1 -- 172.21.15.32:0/1866883012 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(25 keys) ==== 1011+0+0 (unknown 2148541123 0 0) 0x7fc9c8001f00 con 0x7fc9d8106110 2024-04-24T17:27:21.233 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.409+0000 7fc9d4ff9700 1 -- 172.21.15.32:0/1866883012 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fc9c8006230 con 0x7fc9d8106110 2024-04-24T17:27:21.233 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.409+0000 7fc9d4ff9700 1 -- 172.21.15.32:0/1866883012 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc9c8006840 con 0x7fc9d8106110 2024-04-24T17:27:21.233 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.409+0000 7fc9d4ff9700 1 -- 172.21.15.32:0/1866883012 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 2428325519 0 0) 0x7fc9c8007f40 con 0x7fc9d8106110 2024-04-24T17:27:21.233 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.409+0000 7fc9d4ff9700 10 client.14106.objecter ms_dispatch 0x7fc9d8001550 osd_map(1..1 src has 1..1) 2024-04-24T17:27:21.233 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.409+0000 7fc9d4ff9700 3 client.14106.objecter handle_osd_map got epochs [1,1] > 0 2024-04-24T17:27:21.233 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.409+0000 7fc9d4ff9700 3 client.14106.objecter handle_osd_map decoding full epoch 1 2024-04-24T17:27:21.233 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.409+0000 7fc9d4ff9700 20 client.14106.objecter dump_active .. 0 homeless 2024-04-24T17:27:21.233 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.409+0000 7fc9de1d1700 1 -- 172.21.15.32:0/1866883012 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9c4005150 con 0x7fc9d8106110 2024-04-24T17:27:21.233 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.413+0000 7fc9d4ff9700 1 -- 172.21.15.32:0/1866883012 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (unknown 1092875540 0 2268998900) 0x7fc9c00056e0 con 0x7fc9d8106110 2024-04-24T17:27:21.234 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.549+0000 7fc9de1d1700 1 -- 172.21.15.32:0/1866883012 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fc9c4005440 con 0x7fc9d8106110 2024-04-24T17:27:21.234 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.549+0000 7fc9d4ff9700 1 -- 172.21.15.32:0/1866883012 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1240 (unknown 4201413639 0 524151720) 0x7fc9c80059d0 con 0x7fc9d8106110 2024-04-24T17:27:21.234 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.549+0000 7fc9de1d1700 10 librados: watch_flush enter 2024-04-24T17:27:21.234 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.549+0000 7fc9de1d1700 10 librados: watch_flush exit 2024-04-24T17:27:21.234 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.549+0000 7fc9de1d1700 20 client.14106.objecter shutdown clearing up homeless session... 2024-04-24T17:27:21.234 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.549+0000 7fc9de1d1700 10 client.14106.objecter successfully canceled tick 2024-04-24T17:27:21.234 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.549+0000 7fc9de1d1700 1 -- 172.21.15.32:0/1866883012 >> v1:172.21.15.32:6789/0 conn(0x7fc9d8106110 legacy=0x7fc9d819d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:21.234 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.549+0000 7fc9de1d1700 1 -- 172.21.15.32:0/1866883012 shutdown_connections 2024-04-24T17:27:21.234 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.549+0000 7fc9de1d1700 1 -- 172.21.15.32:0/1866883012 >> 172.21.15.32:0/1866883012 conn(0x7fc9d8101cd0 msgr2=0x7fc9d8194320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:27:21.234 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.549+0000 7fc9de1d1700 1 -- 172.21.15.32:0/1866883012 shutdown_connections 2024-04-24T17:27:21.234 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.549+0000 7fc9de1d1700 1 -- 172.21.15.32:0/1866883012 wait complete. 2024-04-24T17:27:21.234 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:20.553+0000 7fc9de1d1700 1 librados: shutdown 2024-04-24T17:27:21.234 INFO:teuthology.orchestra.run.smithi032.stdout:mgr not available, waiting (3/15)... 2024-04-24T17:27:21.501 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:21 smithi032 bash[21712]: debug 2024-04-24T17:27:21.101+0000 7f91b82a8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T17:27:22.001 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:21 smithi032 bash[21457]: cluster 2024-04-24T17:27:21.128804+0000 mon.a (mon.0) 16 : cluster [INF] Activating manager daemon y 2024-04-24T17:27:22.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:21 smithi032 bash[21457]: cluster 2024-04-24T17:27:21.128804+0000 mon.a (mon.0) 16 : cluster [INF] Activating manager daemon y 2024-04-24T17:27:22.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:21 smithi032 bash[21457]: cluster 2024-04-24T17:27:21.136462+0000 mon.a (mon.0) 17 : cluster [DBG] mgrmap e2: y(active, starting, since 0.007934s) 2024-04-24T17:27:22.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:21 smithi032 bash[21457]: cluster 2024-04-24T17:27:21.136462+0000 mon.a (mon.0) 17 : cluster [DBG] mgrmap e2: y(active, starting, since 0.007934s) 2024-04-24T17:27:22.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:21 smithi032 bash[21457]: audit 2024-04-24T17:27:21.137730+0000 mon.a (mon.0) 18 : audit [DBG] from='mgr.14100 v1:172.21.15.32:0/272809882' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-24T17:27:22.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:21 smithi032 bash[21457]: audit 2024-04-24T17:27:21.137730+0000 mon.a (mon.0) 18 : audit [DBG] from='mgr.14100 v1:172.21.15.32:0/272809882' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-24T17:27:22.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:21 smithi032 bash[21457]: audit 2024-04-24T17:27:21.137920+0000 mon.a (mon.0) 19 : audit [DBG] from='mgr.14100 v1:172.21.15.32:0/272809882' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-24T17:27:22.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:21 smithi032 bash[21457]: audit 2024-04-24T17:27:21.137920+0000 mon.a (mon.0) 19 : audit [DBG] from='mgr.14100 v1:172.21.15.32:0/272809882' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-24T17:27:22.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:21 smithi032 bash[21457]: audit 2024-04-24T17:27:21.138051+0000 mon.a (mon.0) 20 : audit [DBG] from='mgr.14100 v1:172.21.15.32:0/272809882' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-24T17:27:22.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:21 smithi032 bash[21457]: audit 2024-04-24T17:27:21.138051+0000 mon.a (mon.0) 20 : audit [DBG] from='mgr.14100 v1:172.21.15.32:0/272809882' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-24T17:27:22.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:21 smithi032 bash[21457]: audit 2024-04-24T17:27:21.138185+0000 mon.a (mon.0) 21 : audit [DBG] from='mgr.14100 v1:172.21.15.32:0/272809882' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-24T17:27:22.003 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:21 smithi032 bash[21457]: audit 2024-04-24T17:27:21.138185+0000 mon.a (mon.0) 21 : audit [DBG] from='mgr.14100 v1:172.21.15.32:0/272809882' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-24T17:27:22.003 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:21 smithi032 bash[21457]: audit 2024-04-24T17:27:21.138318+0000 mon.a (mon.0) 22 : audit [DBG] from='mgr.14100 v1:172.21.15.32:0/272809882' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-24T17:27:22.003 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:21 smithi032 bash[21457]: audit 2024-04-24T17:27:21.138318+0000 mon.a (mon.0) 22 : audit [DBG] from='mgr.14100 v1:172.21.15.32:0/272809882' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-24T17:27:22.003 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:21 smithi032 bash[21457]: audit 2024-04-24T17:27:21.138440+0000 mon.a (mon.0) 23 : audit [DBG] from='mgr.14100 v1:172.21.15.32:0/272809882' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-24T17:27:22.003 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:21 smithi032 bash[21457]: audit 2024-04-24T17:27:21.138440+0000 mon.a (mon.0) 23 : audit [DBG] from='mgr.14100 v1:172.21.15.32:0/272809882' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-24T17:27:22.003 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:21 smithi032 bash[21457]: audit 2024-04-24T17:27:21.138579+0000 mon.a (mon.0) 24 : audit [DBG] from='mgr.14100 v1:172.21.15.32:0/272809882' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T17:27:22.003 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:21 smithi032 bash[21457]: audit 2024-04-24T17:27:21.138579+0000 mon.a (mon.0) 24 : audit [DBG] from='mgr.14100 v1:172.21.15.32:0/272809882' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T17:27:22.003 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:21 smithi032 bash[21457]: audit 2024-04-24T17:27:21.139847+0000 mon.a (mon.0) 25 : audit [DBG] from='mgr.14100 v1:172.21.15.32:0/272809882' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-24T17:27:22.003 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:21 smithi032 bash[21457]: audit 2024-04-24T17:27:21.139847+0000 mon.a (mon.0) 25 : audit [DBG] from='mgr.14100 v1:172.21.15.32:0/272809882' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-24T17:27:22.003 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:21 smithi032 bash[21457]: cluster 2024-04-24T17:27:21.146844+0000 mon.a (mon.0) 26 : cluster [INF] Manager daemon y is now available 2024-04-24T17:27:22.003 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:21 smithi032 bash[21457]: cluster 2024-04-24T17:27:21.146844+0000 mon.a (mon.0) 26 : cluster [INF] Manager daemon y is now available 2024-04-24T17:27:22.003 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:21 smithi032 bash[21457]: audit 2024-04-24T17:27:21.159369+0000 mon.a (mon.0) 27 : audit [INF] from='mgr.14100 v1:172.21.15.32:0/272809882' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T17:27:22.003 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:21 smithi032 bash[21457]: audit 2024-04-24T17:27:21.159369+0000 mon.a (mon.0) 27 : audit [INF] from='mgr.14100 v1:172.21.15.32:0/272809882' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T17:27:22.004 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:21 smithi032 bash[21457]: audit 2024-04-24T17:27:21.162093+0000 mon.a (mon.0) 28 : audit [INF] from='mgr.14100 v1:172.21.15.32:0/272809882' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T17:27:22.004 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:21 smithi032 bash[21457]: audit 2024-04-24T17:27:21.162093+0000 mon.a (mon.0) 28 : audit [INF] from='mgr.14100 v1:172.21.15.32:0/272809882' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T17:27:22.004 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:21 smithi032 bash[21457]: audit 2024-04-24T17:27:21.165409+0000 mon.a (mon.0) 29 : audit [INF] from='mgr.14100 v1:172.21.15.32:0/272809882' entity='mgr.y' 2024-04-24T17:27:22.004 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:21 smithi032 bash[21457]: audit 2024-04-24T17:27:21.165409+0000 mon.a (mon.0) 29 : audit [INF] from='mgr.14100 v1:172.21.15.32:0/272809882' entity='mgr.y' 2024-04-24T17:27:22.004 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:21 smithi032 bash[21457]: audit 2024-04-24T17:27:21.167287+0000 mon.a (mon.0) 30 : audit [INF] from='mgr.14100 v1:172.21.15.32:0/272809882' entity='mgr.y' 2024-04-24T17:27:22.004 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:21 smithi032 bash[21457]: audit 2024-04-24T17:27:21.167287+0000 mon.a (mon.0) 30 : audit [INF] from='mgr.14100 v1:172.21.15.32:0/272809882' entity='mgr.y' 2024-04-24T17:27:22.004 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:21 smithi032 bash[21457]: audit 2024-04-24T17:27:21.168930+0000 mon.a (mon.0) 31 : audit [INF] from='mgr.14100 v1:172.21.15.32:0/272809882' entity='mgr.y' 2024-04-24T17:27:22.004 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:21 smithi032 bash[21457]: audit 2024-04-24T17:27:21.168930+0000 mon.a (mon.0) 31 : audit [INF] from='mgr.14100 v1:172.21.15.32:0/272809882' entity='mgr.y' 2024-04-24T17:27:23.501 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:23 smithi032 bash[21457]: cluster 2024-04-24T17:27:22.144331+0000 mon.a (mon.0) 32 : cluster [DBG] mgrmap e3: y(active, since 1.0158s) 2024-04-24T17:27:23.501 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:23 smithi032 bash[21457]: cluster 2024-04-24T17:27:22.144331+0000 mon.a (mon.0) 32 : cluster [DBG] mgrmap e3: y(active, since 1.0158s) 2024-04-24T17:27:25.351 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-24T17:27:25.351 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-04-24T17:27:25.351 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsid": "a7981b84-025f-11ef-bc93-c7b262605968", 2024-04-24T17:27:25.351 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "health": { 2024-04-24T17:27:25.352 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-24T17:27:25.352 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-24T17:27:25.352 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-24T17:27:25.352 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-24T17:27:25.352 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-24T17:27:25.352 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-24T17:27:25.352 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 0 2024-04-24T17:27:25.352 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-24T17:27:25.352 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-24T17:27:25.352 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "a" 2024-04-24T17:27:25.352 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-24T17:27:25.352 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-04-24T17:27:25.352 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-24T17:27:25.352 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T17:27:25.353 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-24T17:27:25.353 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-24T17:27:25.353 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-24T17:27:25.353 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-24T17:27:25.353 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T17:27:25.353 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-24T17:27:25.353 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-24T17:27:25.353 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-24T17:27:25.362 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-24T17:27:25.363 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-24T17:27:25.363 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-24T17:27:25.363 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-24T17:27:25.363 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-24T17:27:25.363 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-24T17:27:25.363 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-24T17:27:25.363 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-24T17:27:25.363 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-24T17:27:25.363 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-24T17:27:25.363 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-24T17:27:25.363 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-24T17:27:25.364 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-24T17:27:25.364 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-24T17:27:25.364 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-24T17:27:25.364 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T17:27:25.364 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-24T17:27:25.364 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-24T17:27:25.364 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-24T17:27:25.364 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-24T17:27:25.364 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-24T17:27:25.364 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-24T17:27:25.364 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-24T17:27:25.364 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-24T17:27:25.364 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-24T17:27:25.364 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "restful" 2024-04-24T17:27:25.365 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-24T17:27:25.365 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-24T17:27:25.365 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-24T17:27:25.365 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-24T17:27:25.365 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T17:27:25.365 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modified": "2024-04-24T17:27:04.037165+0000", 2024-04-24T17:27:25.365 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-24T17:27:25.365 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-24T17:27:25.365 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-24T17:27:25.365 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-04-24T17:27:25.365 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.109+0000 7f122ba33700 1 Processor -- start 2024-04-24T17:27:25.365 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.109+0000 7f122ba33700 1 -- start start 2024-04-24T17:27:25.365 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.109+0000 7f122ba33700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1224108950 con 0x7f1224107d90 2024-04-24T17:27:25.365 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.109+0000 7f12297cf700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f1224107d90 0x7f12241081b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35068/0 (socket says 172.21.15.32:35068) 2024-04-24T17:27:25.366 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.109+0000 7f12297cf700 1 -- 172.21.15.32:0/730208504 learned_addr learned my addr 172.21.15.32:0/730208504 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:27:25.366 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.109+0000 7f1217fff700 1 -- 172.21.15.32:0/730208504 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 944131588 0 0) 0x7f1224108950 con 0x7f1224107d90 2024-04-24T17:27:25.366 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.109+0000 7f1217fff700 1 -- 172.21.15.32:0/730208504 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12100035b0 con 0x7f1224107d90 2024-04-24T17:27:25.366 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.109+0000 7f1217fff700 1 -- 172.21.15.32:0/730208504 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4260583269 0 0) 0x7f12100035b0 con 0x7f1224107d90 2024-04-24T17:27:25.366 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.109+0000 7f1217fff700 1 -- 172.21.15.32:0/730208504 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1224106930 con 0x7f1224107d90 2024-04-24T17:27:25.366 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.109+0000 7f1217fff700 1 -- 172.21.15.32:0/730208504 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f1218001580 con 0x7f1224107d90 2024-04-24T17:27:25.366 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.109+0000 7f1217fff700 1 -- 172.21.15.32:0/730208504 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(25 keys) ==== 1011+0+0 (unknown 2148541123 0 0) 0x7f1218001c40 con 0x7f1224107d90 2024-04-24T17:27:25.366 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.109+0000 7f1217fff700 1 -- 172.21.15.32:0/730208504 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f1218005960 con 0x7f1224107d90 2024-04-24T17:27:25.366 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.113+0000 7f122ba33700 1 -- 172.21.15.32:0/730208504 >> v1:172.21.15.32:6789/0 conn(0x7f1224107d90 legacy=0x7f12241081b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:25.366 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.113+0000 7f122ba33700 1 -- 172.21.15.32:0/730208504 shutdown_connections 2024-04-24T17:27:25.366 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.113+0000 7f122ba33700 1 -- 172.21.15.32:0/730208504 >> 172.21.15.32:0/730208504 conn(0x7f12241014c0 msgr2=0x7f1224103920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:27:25.366 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.113+0000 7f122ba33700 1 -- 172.21.15.32:0/730208504 shutdown_connections 2024-04-24T17:27:25.366 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.113+0000 7f122ba33700 1 -- 172.21.15.32:0/730208504 wait complete. 2024-04-24T17:27:25.367 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.113+0000 7f122ba33700 1 librados: starting msgr at 2024-04-24T17:27:25.367 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.113+0000 7f122ba33700 1 librados: starting objecter 2024-04-24T17:27:25.367 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.113+0000 7f122ba33700 1 Processor -- start 2024-04-24T17:27:25.367 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.113+0000 7f122ba33700 1 -- start start 2024-04-24T17:27:25.367 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.113+0000 7f122ba33700 1 librados: setting wanted keys 2024-04-24T17:27:25.367 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.113+0000 7f122ba33700 1 librados: calling monclient init 2024-04-24T17:27:25.367 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.113+0000 7f122ba33700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f122419dd10 con 0x7f1224107d90 2024-04-24T17:27:25.367 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.113+0000 7f12297cf700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f1224107d90 0x7f122419d600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35082/0 (socket says 172.21.15.32:35082) 2024-04-24T17:27:25.367 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.113+0000 7f12297cf700 1 -- 172.21.15.32:0/1041484111 learned_addr learned my addr 172.21.15.32:0/1041484111 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:27:25.367 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.113+0000 7f12167fc700 10 client.?.objecter ms_handle_connect 0x7f1224107d90 2024-04-24T17:27:25.367 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.113+0000 7f12167fc700 10 client.?.objecter resend_mon_ops 2024-04-24T17:27:25.367 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.113+0000 7f12167fc700 1 -- 172.21.15.32:0/1041484111 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4127504465 0 0) 0x7f122419dd10 con 0x7f1224107d90 2024-04-24T17:27:25.367 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.113+0000 7f12167fc700 1 -- 172.21.15.32:0/1041484111 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12100035b0 con 0x7f1224107d90 2024-04-24T17:27:25.367 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.113+0000 7f12167fc700 1 -- 172.21.15.32:0/1041484111 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 779009394 0 0) 0x7f12100035b0 con 0x7f1224107d90 2024-04-24T17:27:25.368 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.113+0000 7f12167fc700 1 -- 172.21.15.32:0/1041484111 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f122419dd10 con 0x7f1224107d90 2024-04-24T17:27:25.368 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.113+0000 7f12167fc700 1 -- 172.21.15.32:0/1041484111 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f1218001080 con 0x7f1224107d90 2024-04-24T17:27:25.368 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.113+0000 7f12167fc700 1 -- 172.21.15.32:0/1041484111 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4284843650 0 0) 0x7f122419dd10 con 0x7f1224107d90 2024-04-24T17:27:25.368 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.113+0000 7f12167fc700 1 -- 172.21.15.32:0/1041484111 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f122419dee0 con 0x7f1224107d90 2024-04-24T17:27:25.368 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.113+0000 7f122ba33700 1 -- 172.21.15.32:0/1041484111 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f122419e1d0 con 0x7f1224107d90 2024-04-24T17:27:25.368 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.113+0000 7f122ba33700 10 client.14114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-24T17:27:25.368 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.113+0000 7f122ba33700 1 -- 172.21.15.32:0/1041484111 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f12241a1d20 con 0x7f1224107d90 2024-04-24T17:27:25.368 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.113+0000 7f122ba33700 1 librados: init done 2024-04-24T17:27:25.368 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.113+0000 7f12167fc700 1 -- 172.21.15.32:0/1041484111 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(25 keys) ==== 1011+0+0 (unknown 2148541123 0 0) 0x7f1218005590 con 0x7f1224107d90 2024-04-24T17:27:25.368 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.113+0000 7f12167fc700 1 -- 172.21.15.32:0/1041484111 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f1218005fd0 con 0x7f1224107d90 2024-04-24T17:27:25.368 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.113+0000 7f12167fc700 1 -- 172.21.15.32:0/1041484111 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 3) ==== 46406+0+0 (unknown 2956483143 0 0) 0x7f12180116c0 con 0x7f1224107d90 2024-04-24T17:27:25.368 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.113+0000 7f122ba33700 1 -- 172.21.15.32:0/1041484111 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1208005150 con 0x7f1224107d90 2024-04-24T17:27:25.368 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.113+0000 7f12167fc700 1 -- 172.21.15.32:0/1041484111 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 2428325519 0 0) 0x7f1218047b90 con 0x7f1224107d90 2024-04-24T17:27:25.368 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.113+0000 7f12167fc700 10 client.14114.objecter ms_dispatch 0x7f1224001550 osd_map(1..1 src has 1..1) 2024-04-24T17:27:25.369 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.113+0000 7f12167fc700 3 client.14114.objecter handle_osd_map got epochs [1,1] > 0 2024-04-24T17:27:25.369 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.113+0000 7f12167fc700 3 client.14114.objecter handle_osd_map decoding full epoch 1 2024-04-24T17:27:25.369 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.113+0000 7f12167fc700 20 client.14114.objecter dump_active .. 0 homeless 2024-04-24T17:27:25.369 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.117+0000 7f12167fc700 10 client.14114.objecter ms_handle_connect 0x7f121003c290 2024-04-24T17:27:25.369 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.121+0000 7f12167fc700 1 -- 172.21.15.32:0/1041484111 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188728 (unknown 1092875540 0 2656817166) 0x7f1218017550 con 0x7f1224107d90 2024-04-24T17:27:25.369 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.133+0000 7f12167fc700 1 -- 172.21.15.32:0/1041484111 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mgrmap(e 4) ==== 46512+0+0 (unknown 502058237 0 0) 0x7f1218001a50 con 0x7f1224107d90 2024-04-24T17:27:25.369 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.673+0000 7f122ba33700 1 -- 172.21.15.32:0/1041484111 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f1208005440 con 0x7f1224107d90 2024-04-24T17:27:25.369 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.677+0000 7f12167fc700 1 -- 172.21.15.32:0/1041484111 <== mon.0 v1:172.21.15.32:6789/0 11 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (unknown 4201413639 0 2033571608) 0x7f1218016e50 con 0x7f1224107d90 2024-04-24T17:27:25.369 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.677+0000 7f122ba33700 10 librados: watch_flush enter 2024-04-24T17:27:25.369 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.677+0000 7f122ba33700 10 librados: watch_flush exit 2024-04-24T17:27:25.369 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.677+0000 7f122ba33700 20 client.14114.objecter shutdown clearing up homeless session... 2024-04-24T17:27:25.369 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.677+0000 7f122ba33700 10 client.14114.objecter successfully canceled tick 2024-04-24T17:27:25.369 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.677+0000 7f122ba33700 1 -- 172.21.15.32:0/1041484111 >> v1:172.21.15.32:6800/3443430034 conn(0x7f121003c290 legacy=0x7f121003e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:25.370 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.677+0000 7f122ba33700 1 -- 172.21.15.32:0/1041484111 >> v1:172.21.15.32:6789/0 conn(0x7f1224107d90 legacy=0x7f122419d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:25.370 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.677+0000 7f122ba33700 1 -- 172.21.15.32:0/1041484111 shutdown_connections 2024-04-24T17:27:25.370 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.677+0000 7f122ba33700 1 -- 172.21.15.32:0/1041484111 >> 172.21.15.32:0/1041484111 conn(0x7f12241014c0 msgr2=0x7f1224103920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:27:25.370 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.677+0000 7f122ba33700 1 -- 172.21.15.32:0/1041484111 shutdown_connections 2024-04-24T17:27:25.370 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.681+0000 7f122ba33700 1 -- 172.21.15.32:0/1041484111 wait complete. 2024-04-24T17:27:25.370 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:24.681+0000 7f122ba33700 1 librados: shutdown 2024-04-24T17:27:25.370 INFO:teuthology.orchestra.run.smithi032.stdout:mgr is available 2024-04-24T17:27:25.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:25 smithi032 bash[21457]: cluster 2024-04-24T17:27:24.139729+0000 mon.a (mon.0) 33 : cluster [DBG] mgrmap e4: y(active, since 3s) 2024-04-24T17:27:25.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:25 smithi032 bash[21457]: cluster 2024-04-24T17:27:24.139729+0000 mon.a (mon.0) 33 : cluster [DBG] mgrmap e4: y(active, since 3s) 2024-04-24T17:27:25.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:25 smithi032 bash[21457]: audit 2024-04-24T17:27:24.679537+0000 mon.a (mon.0) 34 : audit [DBG] from='client.? v1:172.21.15.32:0/1041484111' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-24T17:27:25.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:25 smithi032 bash[21457]: audit 2024-04-24T17:27:24.679537+0000 mon.a (mon.0) 34 : audit [DBG] from='client.? v1:172.21.15.32:0/1041484111' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-24T17:27:27.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:26 smithi032 bash[21457]: audit 2024-04-24T17:27:26.739391+0000 mon.a (mon.0) 35 : audit [INF] from='client.? v1:172.21.15.32:0/3055491663' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-24T17:27:27.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:26 smithi032 bash[21457]: audit 2024-04-24T17:27:26.739391+0000 mon.a (mon.0) 35 : audit [INF] from='client.? v1:172.21.15.32:0/3055491663' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-24T17:27:27.331 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-24T17:27:27.331 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout [global] 2024-04-24T17:27:27.331 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout fsid = a7981b84-025f-11ef-bc93-c7b262605968 2024-04-24T17:27:27.331 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-24T17:27:27.331 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-24T17:27:27.331 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-24T17:27:27.331 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-24T17:27:27.331 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-24T17:27:27.332 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-24T17:27:27.337 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-24T17:27:27.337 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-24T17:27:27.337 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-24T17:27:27.337 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout [osd] 2024-04-24T17:27:27.337 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-24T17:27:27.337 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-24T17:27:27.337 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.333+0000 7f746f39e700 1 Processor -- start 2024-04-24T17:27:27.337 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.333+0000 7f746f39e700 1 -- start start 2024-04-24T17:27:27.337 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.333+0000 7f746f39e700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f746807ae00 con 0x7f7468108360 2024-04-24T17:27:27.338 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.333+0000 7f746d13a700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f7468108360 0x7f746810a750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55222/0 (socket says 172.21.15.32:55222) 2024-04-24T17:27:27.338 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.333+0000 7f746d13a700 1 -- 172.21.15.32:0/3124686261 learned_addr learned my addr 172.21.15.32:0/3124686261 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:27:27.338 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.333+0000 7f7467fff700 1 -- 172.21.15.32:0/3124686261 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3489749589 0 0) 0x7f746807ae00 con 0x7f7468108360 2024-04-24T17:27:27.338 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.333+0000 7f7467fff700 1 -- 172.21.15.32:0/3124686261 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74500035b0 con 0x7f7468108360 2024-04-24T17:27:27.338 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.333+0000 7f7467fff700 1 -- 172.21.15.32:0/3124686261 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1645293276 0 0) 0x7f74500035b0 con 0x7f7468108360 2024-04-24T17:27:27.338 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.333+0000 7f7467fff700 1 -- 172.21.15.32:0/3124686261 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f746810ce80 con 0x7f7468108360 2024-04-24T17:27:27.338 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.333+0000 7f7467fff700 1 -- 172.21.15.32:0/3124686261 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f7458001580 con 0x7f7468108360 2024-04-24T17:27:27.338 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.333+0000 7f7467fff700 1 -- 172.21.15.32:0/3124686261 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(25 keys) ==== 1011+0+0 (unknown 2148541123 0 0) 0x7f7458001c40 con 0x7f7468108360 2024-04-24T17:27:27.338 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.333+0000 7f7467fff700 1 -- 172.21.15.32:0/3124686261 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f7458005960 con 0x7f7468108360 2024-04-24T17:27:27.338 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.333+0000 7f746f39e700 1 -- 172.21.15.32:0/3124686261 >> v1:172.21.15.32:6789/0 conn(0x7f7468108360 legacy=0x7f746810a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:27.338 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.333+0000 7f746f39e700 1 -- 172.21.15.32:0/3124686261 shutdown_connections 2024-04-24T17:27:27.339 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.333+0000 7f746f39e700 1 -- 172.21.15.32:0/3124686261 >> 172.21.15.32:0/3124686261 conn(0x7f7468101cd0 msgr2=0x7f7468104150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:27:27.339 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.333+0000 7f746f39e700 1 -- 172.21.15.32:0/3124686261 shutdown_connections 2024-04-24T17:27:27.339 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.333+0000 7f746f39e700 1 -- 172.21.15.32:0/3124686261 wait complete. 2024-04-24T17:27:27.339 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.333+0000 7f746f39e700 1 librados: starting msgr at 2024-04-24T17:27:27.339 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.333+0000 7f746f39e700 1 librados: starting objecter 2024-04-24T17:27:27.339 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.333+0000 7f746f39e700 1 Processor -- start 2024-04-24T17:27:27.339 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.333+0000 7f746f39e700 1 -- start start 2024-04-24T17:27:27.339 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.333+0000 7f746f39e700 1 librados: setting wanted keys 2024-04-24T17:27:27.339 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.333+0000 7f746f39e700 1 librados: calling monclient init 2024-04-24T17:27:27.339 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.333+0000 7f746f39e700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f746819bbe0 con 0x7f7468108360 2024-04-24T17:27:27.339 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.333+0000 7f746d13a700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f7468108360 0x7f746819b4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55230/0 (socket says 172.21.15.32:55230) 2024-04-24T17:27:27.340 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.333+0000 7f746d13a700 1 -- 172.21.15.32:0/3055491663 learned_addr learned my addr 172.21.15.32:0/3055491663 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:27:27.340 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.333+0000 7f74667fc700 10 client.?.objecter ms_handle_connect 0x7f7468108360 2024-04-24T17:27:27.340 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.333+0000 7f74667fc700 10 client.?.objecter resend_mon_ops 2024-04-24T17:27:27.340 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.333+0000 7f74667fc700 1 -- 172.21.15.32:0/3055491663 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1395738571 0 0) 0x7f746819bbe0 con 0x7f7468108360 2024-04-24T17:27:27.340 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.333+0000 7f74667fc700 1 -- 172.21.15.32:0/3055491663 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74500035b0 con 0x7f7468108360 2024-04-24T17:27:27.340 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.337+0000 7f74667fc700 1 -- 172.21.15.32:0/3055491663 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1214291504 0 0) 0x7f74500035b0 con 0x7f7468108360 2024-04-24T17:27:27.340 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.337+0000 7f74667fc700 1 -- 172.21.15.32:0/3055491663 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f746819bbe0 con 0x7f7468108360 2024-04-24T17:27:27.340 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.337+0000 7f74667fc700 1 -- 172.21.15.32:0/3055491663 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f7458001080 con 0x7f7468108360 2024-04-24T17:27:27.344 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.337+0000 7f74667fc700 1 -- 172.21.15.32:0/3055491663 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3508388855 0 0) 0x7f746819bbe0 con 0x7f7468108360 2024-04-24T17:27:27.344 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.337+0000 7f74667fc700 1 -- 172.21.15.32:0/3055491663 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f746819bdb0 con 0x7f7468108360 2024-04-24T17:27:27.344 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.337+0000 7f746f39e700 1 -- 172.21.15.32:0/3055491663 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f746819c0a0 con 0x7f7468108360 2024-04-24T17:27:27.344 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.337+0000 7f746f39e700 10 client.14116.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-24T17:27:27.344 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.337+0000 7f746f39e700 1 -- 172.21.15.32:0/3055491663 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f746819fbf0 con 0x7f7468108360 2024-04-24T17:27:27.344 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.337+0000 7f746f39e700 1 librados: init done 2024-04-24T17:27:27.344 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.337+0000 7f74667fc700 1 -- 172.21.15.32:0/3055491663 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(25 keys) ==== 1011+0+0 (unknown 2148541123 0 0) 0x7f7458005590 con 0x7f7468108360 2024-04-24T17:27:27.344 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.337+0000 7f74667fc700 1 -- 172.21.15.32:0/3055491663 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f7458005fd0 con 0x7f7468108360 2024-04-24T17:27:27.344 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.337+0000 7f74667fc700 1 -- 172.21.15.32:0/3055491663 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) ==== 46512+0+0 (unknown 502058237 0 0) 0x7f7458011770 con 0x7f7468108360 2024-04-24T17:27:27.344 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.337+0000 7f746f39e700 1 -- 172.21.15.32:0/3055491663 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7454005120 con 0x7f7468108360 2024-04-24T17:27:27.344 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.337+0000 7f74667fc700 1 -- 172.21.15.32:0/3055491663 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 2428325519 0 0) 0x7f7458047ee0 con 0x7f7468108360 2024-04-24T17:27:27.345 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.337+0000 7f74667fc700 10 client.14116.objecter ms_dispatch 0x7f7468001550 osd_map(1..1 src has 1..1) 2024-04-24T17:27:27.345 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.337+0000 7f74667fc700 3 client.14116.objecter handle_osd_map got epochs [1,1] > 0 2024-04-24T17:27:27.345 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.337+0000 7f74667fc700 3 client.14116.objecter handle_osd_map decoding full epoch 1 2024-04-24T17:27:27.345 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.337+0000 7f74667fc700 20 client.14116.objecter dump_active .. 0 homeless 2024-04-24T17:27:27.345 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.337+0000 7f74667fc700 10 client.14116.objecter ms_handle_connect 0x7f745003c2f0 2024-04-24T17:27:27.345 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.341+0000 7f74667fc700 1 -- 172.21.15.32:0/3055491663 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188728 (unknown 1092875540 0 2656817166) 0x7f7458017670 con 0x7f7468108360 2024-04-24T17:27:27.345 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.737+0000 7f746f39e700 1 -- 172.21.15.32:0/3055491663 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f7454002400 con 0x7f7468108360 2024-04-24T17:27:27.345 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.737+0000 7f74667fc700 1 -- 172.21.15.32:0/3055491663 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+318 (unknown 1187553405 0 1690012175) 0x7f7458016f70 con 0x7f7468108360 2024-04-24T17:27:27.345 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.737+0000 7f746f39e700 10 librados: watch_flush enter 2024-04-24T17:27:27.345 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.737+0000 7f746f39e700 10 librados: watch_flush exit 2024-04-24T17:27:27.345 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.737+0000 7f746f39e700 20 client.14116.objecter shutdown clearing up homeless session... 2024-04-24T17:27:27.346 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.737+0000 7f746f39e700 10 client.14116.objecter successfully canceled tick 2024-04-24T17:27:27.346 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.741+0000 7f746f39e700 1 -- 172.21.15.32:0/3055491663 >> v1:172.21.15.32:6800/3443430034 conn(0x7f745003c2f0 legacy=0x7f745003e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:27.346 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.741+0000 7f746f39e700 1 -- 172.21.15.32:0/3055491663 >> v1:172.21.15.32:6789/0 conn(0x7f7468108360 legacy=0x7f746819b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:27.346 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.741+0000 7f746f39e700 1 -- 172.21.15.32:0/3055491663 shutdown_connections 2024-04-24T17:27:27.346 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.741+0000 7f746f39e700 1 -- 172.21.15.32:0/3055491663 >> 172.21.15.32:0/3055491663 conn(0x7f7468101cd0 msgr2=0x7f7468192140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:27:27.346 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.741+0000 7f746f39e700 1 -- 172.21.15.32:0/3055491663 shutdown_connections 2024-04-24T17:27:27.346 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.741+0000 7f746f39e700 1 -- 172.21.15.32:0/3055491663 wait complete. 2024-04-24T17:27:27.346 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:26.741+0000 7f746f39e700 1 librados: shutdown 2024-04-24T17:27:27.346 INFO:teuthology.orchestra.run.smithi032.stdout:Enabling cephadm module... 2024-04-24T17:27:28.831 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:28 smithi032 bash[21457]: audit 2024-04-24T17:27:28.731180+0000 mon.a (mon.0) 36 : audit [INF] from='client.? v1:172.21.15.32:0/610843268' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-24T17:27:28.831 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:28 smithi032 bash[21457]: audit 2024-04-24T17:27:28.731180+0000 mon.a (mon.0) 36 : audit [INF] from='client.? v1:172.21.15.32:0/610843268' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-24T17:27:29.251 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:28 smithi032 bash[21712]: ignoring --setuser ceph since I am not root 2024-04-24T17:27:29.251 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:28 smithi032 bash[21712]: ignoring --setgroup ceph since I am not root 2024-04-24T17:27:29.251 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:28 smithi032 bash[21712]: debug 2024-04-24T17:27:28.989+0000 7f48d12fd200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T17:27:29.459 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.205+0000 7fa049405700 1 Processor -- start 2024-04-24T17:27:29.461 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.205+0000 7fa049405700 1 -- start start 2024-04-24T17:27:29.461 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.205+0000 7fa049405700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa04407adc0 con 0x7fa0441083f0 2024-04-24T17:27:29.461 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.205+0000 7fa042ffd700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa0441083f0 0x7fa04410a7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55234/0 (socket says 172.21.15.32:55234) 2024-04-24T17:27:29.461 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.205+0000 7fa042ffd700 1 -- 172.21.15.32:0/2187905564 learned_addr learned my addr 172.21.15.32:0/2187905564 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:27:29.461 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.209+0000 7fa041ffb700 1 -- 172.21.15.32:0/2187905564 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1756837410 0 0) 0x7fa04407adc0 con 0x7fa0441083f0 2024-04-24T17:27:29.461 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.209+0000 7fa041ffb700 1 -- 172.21.15.32:0/2187905564 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa02c0035b0 con 0x7fa0441083f0 2024-04-24T17:27:29.461 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.209+0000 7fa041ffb700 1 -- 172.21.15.32:0/2187905564 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4045909105 0 0) 0x7fa02c0035b0 con 0x7fa0441083f0 2024-04-24T17:27:29.461 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.209+0000 7fa041ffb700 1 -- 172.21.15.32:0/2187905564 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa04410cf10 con 0x7fa0441083f0 2024-04-24T17:27:29.461 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.209+0000 7fa041ffb700 1 -- 172.21.15.32:0/2187905564 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fa034001580 con 0x7fa0441083f0 2024-04-24T17:27:29.461 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.209+0000 7fa041ffb700 1 -- 172.21.15.32:0/2187905564 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(25 keys) ==== 1011+0+0 (unknown 2148541123 0 0) 0x7fa034001c40 con 0x7fa0441083f0 2024-04-24T17:27:29.461 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.209+0000 7fa041ffb700 1 -- 172.21.15.32:0/2187905564 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fa034005960 con 0x7fa0441083f0 2024-04-24T17:27:29.462 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.209+0000 7fa049405700 1 -- 172.21.15.32:0/2187905564 >> v1:172.21.15.32:6789/0 conn(0x7fa0441083f0 legacy=0x7fa04410a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:29.462 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.209+0000 7fa049405700 1 -- 172.21.15.32:0/2187905564 shutdown_connections 2024-04-24T17:27:29.468 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.209+0000 7fa049405700 1 -- 172.21.15.32:0/2187905564 >> 172.21.15.32:0/2187905564 conn(0x7fa044101d60 msgr2=0x7fa0441041a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:27:29.468 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.209+0000 7fa049405700 1 -- 172.21.15.32:0/2187905564 shutdown_connections 2024-04-24T17:27:29.468 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.209+0000 7fa049405700 1 -- 172.21.15.32:0/2187905564 wait complete. 2024-04-24T17:27:29.468 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.209+0000 7fa049405700 1 librados: starting msgr at 2024-04-24T17:27:29.468 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.209+0000 7fa049405700 1 librados: starting objecter 2024-04-24T17:27:29.468 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.209+0000 7fa049405700 1 Processor -- start 2024-04-24T17:27:29.468 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.209+0000 7fa049405700 1 -- start start 2024-04-24T17:27:29.469 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.209+0000 7fa049405700 1 librados: setting wanted keys 2024-04-24T17:27:29.469 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.209+0000 7fa049405700 1 librados: calling monclient init 2024-04-24T17:27:29.469 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.209+0000 7fa049405700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa0441a14a0 con 0x7fa0441083f0 2024-04-24T17:27:29.469 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.209+0000 7fa042ffd700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa0441083f0 0x7fa044107c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55244/0 (socket says 172.21.15.32:55244) 2024-04-24T17:27:29.469 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.209+0000 7fa042ffd700 1 -- 172.21.15.32:0/610843268 learned_addr learned my addr 172.21.15.32:0/610843268 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:27:29.469 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.209+0000 7fa02bfff700 10 client.?.objecter ms_handle_connect 0x7fa0441083f0 2024-04-24T17:27:29.469 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.209+0000 7fa02bfff700 10 client.?.objecter resend_mon_ops 2024-04-24T17:27:29.469 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.209+0000 7fa02bfff700 1 -- 172.21.15.32:0/610843268 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 318059764 0 0) 0x7fa0441a14a0 con 0x7fa0441083f0 2024-04-24T17:27:29.469 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.209+0000 7fa02bfff700 1 -- 172.21.15.32:0/610843268 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa02c0035b0 con 0x7fa0441083f0 2024-04-24T17:27:29.469 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.209+0000 7fa02bfff700 1 -- 172.21.15.32:0/610843268 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3981979861 0 0) 0x7fa02c0035b0 con 0x7fa0441083f0 2024-04-24T17:27:29.469 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.209+0000 7fa02bfff700 1 -- 172.21.15.32:0/610843268 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa0441a14a0 con 0x7fa0441083f0 2024-04-24T17:27:29.469 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.209+0000 7fa02bfff700 1 -- 172.21.15.32:0/610843268 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fa034001080 con 0x7fa0441083f0 2024-04-24T17:27:29.469 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.209+0000 7fa02bfff700 1 -- 172.21.15.32:0/610843268 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 828187495 0 0) 0x7fa0441a14a0 con 0x7fa0441083f0 2024-04-24T17:27:29.469 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.209+0000 7fa02bfff700 1 -- 172.21.15.32:0/610843268 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0441a2680 con 0x7fa0441083f0 2024-04-24T17:27:29.469 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.209+0000 7fa049405700 1 -- 172.21.15.32:0/610843268 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa0441a16d0 con 0x7fa0441083f0 2024-04-24T17:27:29.470 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.209+0000 7fa049405700 10 client.14118.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-24T17:27:29.470 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.209+0000 7fa049405700 1 -- 172.21.15.32:0/610843268 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa0441a1c10 con 0x7fa0441083f0 2024-04-24T17:27:29.470 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.209+0000 7fa049405700 1 librados: init done 2024-04-24T17:27:29.470 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.209+0000 7fa02bfff700 1 -- 172.21.15.32:0/610843268 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(25 keys) ==== 1011+0+0 (unknown 2148541123 0 0) 0x7fa034005590 con 0x7fa0441083f0 2024-04-24T17:27:29.470 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.209+0000 7fa02bfff700 1 -- 172.21.15.32:0/610843268 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fa034005fd0 con 0x7fa0441083f0 2024-04-24T17:27:29.470 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.213+0000 7fa02bfff700 1 -- 172.21.15.32:0/610843268 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) ==== 46512+0+0 (unknown 502058237 0 0) 0x7fa0340117b0 con 0x7fa0441083f0 2024-04-24T17:27:29.470 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.213+0000 7fa049405700 1 -- 172.21.15.32:0/610843268 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa030005150 con 0x7fa0441083f0 2024-04-24T17:27:29.470 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.213+0000 7fa02bfff700 1 -- 172.21.15.32:0/610843268 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 2428325519 0 0) 0x7fa034048020 con 0x7fa0441083f0 2024-04-24T17:27:29.470 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.213+0000 7fa02bfff700 10 client.14118.objecter ms_dispatch 0x7fa044001550 osd_map(1..1 src has 1..1) 2024-04-24T17:27:29.470 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.213+0000 7fa02bfff700 3 client.14118.objecter handle_osd_map got epochs [1,1] > 0 2024-04-24T17:27:29.470 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.213+0000 7fa02bfff700 3 client.14118.objecter handle_osd_map decoding full epoch 1 2024-04-24T17:27:29.470 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.213+0000 7fa02bfff700 20 client.14118.objecter dump_active .. 0 homeless 2024-04-24T17:27:29.470 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.213+0000 7fa02bfff700 10 client.14118.objecter ms_handle_connect 0x7fa02c03c3c0 2024-04-24T17:27:29.471 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.217+0000 7fa02bfff700 1 -- 172.21.15.32:0/610843268 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188728 (unknown 1092875540 0 2656817166) 0x7fa034017690 con 0x7fa0441083f0 2024-04-24T17:27:29.471 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.725+0000 7fa049405700 1 -- 172.21.15.32:0/610843268 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7fa030005440 con 0x7fa0441083f0 2024-04-24T17:27:29.471 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.781+0000 7fa02bfff700 1 -- 172.21.15.32:0/610843268 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (unknown 2263024820 0 0) 0x7fa034016f90 con 0x7fa0441083f0 2024-04-24T17:27:29.471 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.781+0000 7fa049405700 10 librados: watch_flush enter 2024-04-24T17:27:29.471 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.781+0000 7fa02bfff700 1 -- 172.21.15.32:0/610843268 <== mon.0 v1:172.21.15.32:6789/0 11 ==== mgrmap(e 5) ==== 46523+0+0 (unknown 235251547 0 0) 0x7fa034001a50 con 0x7fa0441083f0 2024-04-24T17:27:29.471 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.785+0000 7fa049405700 10 librados: watch_flush exit 2024-04-24T17:27:29.471 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.785+0000 7fa049405700 20 client.14118.objecter shutdown clearing up homeless session... 2024-04-24T17:27:29.471 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.785+0000 7fa049405700 10 client.14118.objecter successfully canceled tick 2024-04-24T17:27:29.471 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.785+0000 7fa049405700 1 -- 172.21.15.32:0/610843268 >> v1:172.21.15.32:6800/3443430034 conn(0x7fa02c03c3c0 legacy=0x7fa02c03e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:29.471 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.785+0000 7fa049405700 1 -- 172.21.15.32:0/610843268 >> v1:172.21.15.32:6789/0 conn(0x7fa0441083f0 legacy=0x7fa044107c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:29.471 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.785+0000 7fa049405700 1 -- 172.21.15.32:0/610843268 shutdown_connections 2024-04-24T17:27:29.471 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.785+0000 7fa049405700 1 -- 172.21.15.32:0/610843268 >> 172.21.15.32:0/610843268 conn(0x7fa044101d60 msgr2=0x7fa044192190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:27:29.471 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.785+0000 7fa049405700 1 -- 172.21.15.32:0/610843268 shutdown_connections 2024-04-24T17:27:29.471 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.785+0000 7fa049405700 1 -- 172.21.15.32:0/610843268 wait complete. 2024-04-24T17:27:29.472 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:28.785+0000 7fa049405700 1 librados: shutdown 2024-04-24T17:27:30.148 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:29 smithi032 bash[21457]: audit 2024-04-24T17:27:28.783135+0000 mon.a (mon.0) 37 : audit [INF] from='client.? v1:172.21.15.32:0/610843268' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-24T17:27:30.148 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:29 smithi032 bash[21457]: audit 2024-04-24T17:27:28.783135+0000 mon.a (mon.0) 37 : audit [INF] from='client.? v1:172.21.15.32:0/610843268' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-24T17:27:30.149 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:29 smithi032 bash[21457]: cluster 2024-04-24T17:27:28.788267+0000 mon.a (mon.0) 38 : cluster [DBG] mgrmap e5: y(active, since 7s) 2024-04-24T17:27:30.149 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:29 smithi032 bash[21457]: cluster 2024-04-24T17:27:28.788267+0000 mon.a (mon.0) 38 : cluster [DBG] mgrmap e5: y(active, since 7s) 2024-04-24T17:27:30.885 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:30 smithi032 bash[21712]: debug 2024-04-24T17:27:30.629+0000 7f48d12fd200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T17:27:30.885 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:30 smithi032 bash[21712]: debug 2024-04-24T17:27:30.729+0000 7f48d12fd200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T17:27:31.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:30 smithi032 bash[21457]: audit 2024-04-24T17:27:30.842376+0000 mon.a (mon.0) 39 : audit [DBG] from='client.? v1:172.21.15.32:0/1506729746' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-24T17:27:31.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:30 smithi032 bash[21457]: audit 2024-04-24T17:27:30.842376+0000 mon.a (mon.0) 39 : audit [DBG] from='client.? v1:172.21.15.32:0/1506729746' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-24T17:27:31.251 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:31 smithi032 bash[21712]: debug 2024-04-24T17:27:31.001+0000 7f48d12fd200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T17:27:31.499 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-04-24T17:27:31.499 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-24T17:27:31.499 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-24T17:27:31.500 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-04-24T17:27:31.500 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-24T17:27:31.500 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-04-24T17:27:31.500 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.321+0000 7f04dc3ac700 1 Processor -- start 2024-04-24T17:27:31.500 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.321+0000 7f04dc3ac700 1 -- start start 2024-04-24T17:27:31.500 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.325+0000 7f04dc3ac700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f04d407ae20 con 0x7f04d4108340 2024-04-24T17:27:31.500 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.325+0000 7f04da148700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f04d4108340 0x7f04d410a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55270/0 (socket says 172.21.15.32:55270) 2024-04-24T17:27:31.500 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.325+0000 7f04da148700 1 -- 172.21.15.32:0/1030343315 learned_addr learned my addr 172.21.15.32:0/1030343315 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:27:31.500 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.325+0000 7f04d9146700 1 -- 172.21.15.32:0/1030343315 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1269881417 0 0) 0x7f04d407ae20 con 0x7f04d4108340 2024-04-24T17:27:31.500 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.325+0000 7f04d9146700 1 -- 172.21.15.32:0/1030343315 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04c00035b0 con 0x7f04d4108340 2024-04-24T17:27:31.500 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.325+0000 7f04d9146700 1 -- 172.21.15.32:0/1030343315 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1231313226 0 0) 0x7f04c00035b0 con 0x7f04d4108340 2024-04-24T17:27:31.500 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.325+0000 7f04d9146700 1 -- 172.21.15.32:0/1030343315 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04d410ce60 con 0x7f04d4108340 2024-04-24T17:27:31.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.325+0000 7f04d9146700 1 -- 172.21.15.32:0/1030343315 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f04c4001580 con 0x7f04d4108340 2024-04-24T17:27:31.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.325+0000 7f04d9146700 1 -- 172.21.15.32:0/1030343315 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(25 keys) ==== 1011+0+0 (unknown 2148541123 0 0) 0x7f04c4001d10 con 0x7f04d4108340 2024-04-24T17:27:31.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.325+0000 7f04d9146700 1 -- 172.21.15.32:0/1030343315 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f04c4005960 con 0x7f04d4108340 2024-04-24T17:27:31.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.325+0000 7f04dc3ac700 1 -- 172.21.15.32:0/1030343315 >> v1:172.21.15.32:6789/0 conn(0x7f04d4108340 legacy=0x7f04d410a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:31.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.325+0000 7f04dc3ac700 1 -- 172.21.15.32:0/1030343315 shutdown_connections 2024-04-24T17:27:31.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.325+0000 7f04dc3ac700 1 -- 172.21.15.32:0/1030343315 >> 172.21.15.32:0/1030343315 conn(0x7f04d4101cd0 msgr2=0x7f04d4104150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:27:31.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.325+0000 7f04dc3ac700 1 -- 172.21.15.32:0/1030343315 shutdown_connections 2024-04-24T17:27:31.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.325+0000 7f04dc3ac700 1 -- 172.21.15.32:0/1030343315 wait complete. 2024-04-24T17:27:31.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.325+0000 7f04dc3ac700 1 librados: starting msgr at 2024-04-24T17:27:31.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.325+0000 7f04dc3ac700 1 librados: starting objecter 2024-04-24T17:27:31.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.325+0000 7f04dc3ac700 1 Processor -- start 2024-04-24T17:27:31.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.325+0000 7f04dc3ac700 1 -- start start 2024-04-24T17:27:31.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.325+0000 7f04dc3ac700 1 librados: setting wanted keys 2024-04-24T17:27:31.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.325+0000 7f04dc3ac700 1 librados: calling monclient init 2024-04-24T17:27:31.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.325+0000 7f04dc3ac700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f04d41a1520 con 0x7f04d4108340 2024-04-24T17:27:31.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.325+0000 7f04da148700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f04d4108340 0x7f04d4107bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55284/0 (socket says 172.21.15.32:55284) 2024-04-24T17:27:31.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.325+0000 7f04da148700 1 -- 172.21.15.32:0/1506729746 learned_addr learned my addr 172.21.15.32:0/1506729746 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:27:31.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.325+0000 7f04cb7fe700 10 client.?.objecter ms_handle_connect 0x7f04d4108340 2024-04-24T17:27:31.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.325+0000 7f04cb7fe700 10 client.?.objecter resend_mon_ops 2024-04-24T17:27:31.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.325+0000 7f04cb7fe700 1 -- 172.21.15.32:0/1506729746 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1152956459 0 0) 0x7f04d41a1520 con 0x7f04d4108340 2024-04-24T17:27:31.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.325+0000 7f04cb7fe700 1 -- 172.21.15.32:0/1506729746 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04c00056e0 con 0x7f04d4108340 2024-04-24T17:27:31.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.325+0000 7f04cb7fe700 1 -- 172.21.15.32:0/1506729746 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 488578204 0 0) 0x7f04d410ce60 con 0x7f04d4108340 2024-04-24T17:27:31.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.325+0000 7f04cb7fe700 1 -- 172.21.15.32:0/1506729746 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f04d41a1520 con 0x7f04d4108340 2024-04-24T17:27:31.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.325+0000 7f04cb7fe700 1 -- 172.21.15.32:0/1506729746 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f04c4004870 con 0x7f04d4108340 2024-04-24T17:27:31.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.325+0000 7f04cb7fe700 1 -- 172.21.15.32:0/1506729746 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2545829219 0 0) 0x7f04d41a1520 con 0x7f04d4108340 2024-04-24T17:27:31.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.325+0000 7f04cb7fe700 1 -- 172.21.15.32:0/1506729746 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04d41a2700 con 0x7f04d4108340 2024-04-24T17:27:31.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.329+0000 7f04dc3ac700 1 -- 172.21.15.32:0/1506729746 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f04d41a1750 con 0x7f04d4108340 2024-04-24T17:27:31.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.329+0000 7f04dc3ac700 10 client.14122.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-24T17:27:31.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.329+0000 7f04dc3ac700 1 -- 172.21.15.32:0/1506729746 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f04d41a1c90 con 0x7f04d4108340 2024-04-24T17:27:31.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.329+0000 7f04dc3ac700 1 librados: init done 2024-04-24T17:27:31.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.329+0000 7f04cb7fe700 1 -- 172.21.15.32:0/1506729746 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(25 keys) ==== 1011+0+0 (unknown 2148541123 0 0) 0x7f04c4001b80 con 0x7f04d4108340 2024-04-24T17:27:31.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.329+0000 7f04cb7fe700 1 -- 172.21.15.32:0/1506729746 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f04c4005eb0 con 0x7f04d4108340 2024-04-24T17:27:31.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.329+0000 7f04cb7fe700 1 -- 172.21.15.32:0/1506729746 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 5) ==== 46523+0+0 (unknown 235251547 0 0) 0x7f04c4011640 con 0x7f04d4108340 2024-04-24T17:27:31.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.329+0000 7f04dc3ac700 1 -- 172.21.15.32:0/1506729746 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04b8005120 con 0x7f04d4108340 2024-04-24T17:27:31.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.329+0000 7f04cb7fe700 1 -- 172.21.15.32:0/1506729746 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 2428325519 0 0) 0x7f04c4047e20 con 0x7f04d4108340 2024-04-24T17:27:31.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.329+0000 7f04cb7fe700 10 client.14122.objecter ms_dispatch 0x7f04d4001550 osd_map(1..1 src has 1..1) 2024-04-24T17:27:31.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.329+0000 7f04cb7fe700 3 client.14122.objecter handle_osd_map got epochs [1,1] > 0 2024-04-24T17:27:31.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.329+0000 7f04cb7fe700 3 client.14122.objecter handle_osd_map decoding full epoch 1 2024-04-24T17:27:31.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.329+0000 7f04d9947700 1 -- 172.21.15.32:0/1506729746 >> v1:172.21.15.32:6800/3443430034 conn(0x7f04c003c270 legacy=0x7f04c003e730 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.32:6800/3443430034 2024-04-24T17:27:31.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.329+0000 7f04cb7fe700 20 client.14122.objecter dump_active .. 0 homeless 2024-04-24T17:27:31.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.333+0000 7f04cb7fe700 1 -- 172.21.15.32:0/1506729746 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188728 (unknown 1092875540 0 2656817166) 0x7f04c00056e0 con 0x7f04d4108340 2024-04-24T17:27:31.504 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.533+0000 7f04d9947700 1 -- 172.21.15.32:0/1506729746 >> v1:172.21.15.32:6800/3443430034 conn(0x7f04c003c270 legacy=0x7f04c003e730 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.32:6800/3443430034 2024-04-24T17:27:31.504 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.837+0000 7f04dc3ac700 1 -- 172.21.15.32:0/1506729746 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f04b8005c20 con 0x7f04d4108340 2024-04-24T17:27:31.504 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.837+0000 7f04cb7fe700 1 -- 172.21.15.32:0/1506729746 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (unknown 3768197548 0 15966916) 0x7f04c4017590 con 0x7f04d4108340 2024-04-24T17:27:31.504 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.841+0000 7f04dc3ac700 10 librados: watch_flush enter 2024-04-24T17:27:31.504 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.841+0000 7f04dc3ac700 10 librados: watch_flush exit 2024-04-24T17:27:31.504 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.841+0000 7f04dc3ac700 20 client.14122.objecter shutdown clearing up homeless session... 2024-04-24T17:27:31.504 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.841+0000 7f04dc3ac700 10 client.14122.objecter successfully canceled tick 2024-04-24T17:27:31.504 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.841+0000 7f04dc3ac700 1 -- 172.21.15.32:0/1506729746 >> v1:172.21.15.32:6800/3443430034 conn(0x7f04c003c270 legacy=0x7f04c003e730 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T17:27:31.504 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.841+0000 7f04dc3ac700 1 -- 172.21.15.32:0/1506729746 >> v1:172.21.15.32:6789/0 conn(0x7f04d4108340 legacy=0x7f04d4107bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:31.504 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.841+0000 7f04dc3ac700 1 -- 172.21.15.32:0/1506729746 shutdown_connections 2024-04-24T17:27:31.504 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.841+0000 7f04dc3ac700 1 -- 172.21.15.32:0/1506729746 >> 172.21.15.32:0/1506729746 conn(0x7f04d4101cd0 msgr2=0x7f04d4192140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:27:31.504 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.841+0000 7f04dc3ac700 1 -- 172.21.15.32:0/1506729746 shutdown_connections 2024-04-24T17:27:31.504 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.841+0000 7f04dc3ac700 1 -- 172.21.15.32:0/1506729746 wait complete. 2024-04-24T17:27:31.504 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:30.841+0000 7f04dc3ac700 1 librados: shutdown 2024-04-24T17:27:31.504 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for the mgr to restart... 2024-04-24T17:27:31.505 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for mgr epoch 5... 2024-04-24T17:27:32.131 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:31 smithi032 bash[21712]: debug 2024-04-24T17:27:31.833+0000 7f48d12fd200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T17:27:32.131 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:31 smithi032 bash[21712]: debug 2024-04-24T17:27:31.949+0000 7f48d12fd200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T17:27:32.131 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:32 smithi032 bash[21712]: debug 2024-04-24T17:27:32.057+0000 7f48d12fd200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T17:27:32.751 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:32 smithi032 bash[21712]: debug 2024-04-24T17:27:32.425+0000 7f48d12fd200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-24T17:27:32.751 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:32 smithi032 bash[21712]: debug 2024-04-24T17:27:32.541+0000 7f48d12fd200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T17:27:32.751 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:32 smithi032 bash[21712]: debug 2024-04-24T17:27:32.649+0000 7f48d12fd200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T17:27:33.501 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:33 smithi032 bash[21712]: debug 2024-04-24T17:27:33.021+0000 7f48d12fd200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T17:27:33.501 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:33 smithi032 bash[21712]: debug 2024-04-24T17:27:33.265+0000 7f48d12fd200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T17:27:33.938 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:33 smithi032 bash[21712]: debug 2024-04-24T17:27:33.537+0000 7f48d12fd200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T17:27:33.938 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:33 smithi032 bash[21712]: debug 2024-04-24T17:27:33.653+0000 7f48d12fd200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T17:27:34.251 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:33 smithi032 bash[21712]: debug 2024-04-24T17:27:33.937+0000 7f48d12fd200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T17:27:34.732 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:34 smithi032 bash[21712]: debug 2024-04-24T17:27:34.361+0000 7f48d12fd200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T17:27:34.732 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:34 smithi032 bash[21712]: debug 2024-04-24T17:27:34.605+0000 7f48d12fd200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T17:27:34.990 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:34 smithi032 bash[21712]: debug 2024-04-24T17:27:34.729+0000 7f48d12fd200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T17:27:35.251 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:34 smithi032 bash[21712]: debug 2024-04-24T17:27:34.989+0000 7f48d12fd200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T17:27:36.458 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:36 smithi032 bash[21712]: debug 2024-04-24T17:27:36.029+0000 7f48d12fd200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T17:27:36.458 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:36 smithi032 bash[21712]: debug 2024-04-24T17:27:36.149+0000 7f48d12fd200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T17:27:36.751 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:36 smithi032 bash[21712]: debug 2024-04-24T17:27:36.457+0000 7f48d12fd200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T17:27:36.751 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:36 smithi032 bash[21712]: debug 2024-04-24T17:27:36.573+0000 7f48d12fd200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T17:27:37.501 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:37 smithi032 bash[21712]: debug 2024-04-24T17:27:37.045+0000 7f48d12fd200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T17:27:37.501 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:37 smithi032 bash[21712]: debug 2024-04-24T17:27:37.169+0000 7f48d12fd200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T17:27:37.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:37 smithi032 bash[21457]: cluster 2024-04-24T17:27:37.192008+0000 mon.a (mon.0) 40 : cluster [INF] Active manager daemon y restarted 2024-04-24T17:27:37.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:37 smithi032 bash[21457]: cluster 2024-04-24T17:27:37.192008+0000 mon.a (mon.0) 40 : cluster [INF] Active manager daemon y restarted 2024-04-24T17:27:37.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:37 smithi032 bash[21457]: cluster 2024-04-24T17:27:37.192463+0000 mon.a (mon.0) 41 : cluster [INF] Activating manager daemon y 2024-04-24T17:27:37.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:37 smithi032 bash[21457]: cluster 2024-04-24T17:27:37.192463+0000 mon.a (mon.0) 41 : cluster [INF] Activating manager daemon y 2024-04-24T17:27:37.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:37 smithi032 bash[21457]: cluster 2024-04-24T17:27:37.196266+0000 mon.a (mon.0) 42 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-04-24T17:27:37.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:37 smithi032 bash[21457]: cluster 2024-04-24T17:27:37.196266+0000 mon.a (mon.0) 42 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-04-24T17:27:37.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:37 smithi032 bash[21457]: cluster 2024-04-24T17:27:37.196387+0000 mon.a (mon.0) 43 : cluster [DBG] mgrmap e6: y(active, starting, since 0.00411238s) 2024-04-24T17:27:37.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:37 smithi032 bash[21457]: cluster 2024-04-24T17:27:37.196387+0000 mon.a (mon.0) 43 : cluster [DBG] mgrmap e6: y(active, starting, since 0.00411238s) 2024-04-24T17:27:37.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:37 smithi032 bash[21457]: audit 2024-04-24T17:27:37.199826+0000 mon.a (mon.0) 44 : audit [DBG] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T17:27:37.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:37 smithi032 bash[21457]: audit 2024-04-24T17:27:37.199826+0000 mon.a (mon.0) 44 : audit [DBG] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T17:27:37.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:37 smithi032 bash[21457]: audit 2024-04-24T17:27:37.201463+0000 mon.a (mon.0) 45 : audit [DBG] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-24T17:27:37.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:37 smithi032 bash[21457]: audit 2024-04-24T17:27:37.201463+0000 mon.a (mon.0) 45 : audit [DBG] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-24T17:27:37.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:37 smithi032 bash[21457]: audit 2024-04-24T17:27:37.202630+0000 mon.a (mon.0) 46 : audit [DBG] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-24T17:27:37.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:37 smithi032 bash[21457]: audit 2024-04-24T17:27:37.202630+0000 mon.a (mon.0) 46 : audit [DBG] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-24T17:27:37.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:37 smithi032 bash[21457]: audit 2024-04-24T17:27:37.202827+0000 mon.a (mon.0) 47 : audit [DBG] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-24T17:27:37.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:37 smithi032 bash[21457]: audit 2024-04-24T17:27:37.202827+0000 mon.a (mon.0) 47 : audit [DBG] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-24T17:27:37.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:37 smithi032 bash[21457]: audit 2024-04-24T17:27:37.203017+0000 mon.a (mon.0) 48 : audit [DBG] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-24T17:27:37.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:37 smithi032 bash[21457]: audit 2024-04-24T17:27:37.203017+0000 mon.a (mon.0) 48 : audit [DBG] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-24T17:27:37.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:37 smithi032 bash[21457]: cluster 2024-04-24T17:27:37.232643+0000 mon.a (mon.0) 49 : cluster [INF] Manager daemon y is now available 2024-04-24T17:27:37.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:37 smithi032 bash[21457]: cluster 2024-04-24T17:27:37.232643+0000 mon.a (mon.0) 49 : cluster [INF] Manager daemon y is now available 2024-04-24T17:27:38.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:38 smithi032 bash[21457]: cephadm 2024-04-24T17:27:37.252528+0000 mgr.y (mgr.14120) 1 : cephadm [INF] Found migration_current of "None". Setting to last migration. 2024-04-24T17:27:38.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:38 smithi032 bash[21457]: cephadm 2024-04-24T17:27:37.252528+0000 mgr.y (mgr.14120) 1 : cephadm [INF] Found migration_current of "None". Setting to last migration. 2024-04-24T17:27:38.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:38 smithi032 bash[21457]: audit 2024-04-24T17:27:37.254499+0000 mon.a (mon.0) 50 : audit [INF] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' 2024-04-24T17:27:38.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:38 smithi032 bash[21457]: audit 2024-04-24T17:27:37.254499+0000 mon.a (mon.0) 50 : audit [INF] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' 2024-04-24T17:27:38.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:38 smithi032 bash[21457]: audit 2024-04-24T17:27:37.257033+0000 mon.a (mon.0) 51 : audit [INF] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' 2024-04-24T17:27:38.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:38 smithi032 bash[21457]: audit 2024-04-24T17:27:37.257033+0000 mon.a (mon.0) 51 : audit [INF] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' 2024-04-24T17:27:38.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:38 smithi032 bash[21457]: audit 2024-04-24T17:27:37.261565+0000 mon.a (mon.0) 52 : audit [DBG] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T17:27:38.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:38 smithi032 bash[21457]: audit 2024-04-24T17:27:37.261565+0000 mon.a (mon.0) 52 : audit [DBG] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T17:27:38.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:38 smithi032 bash[21457]: audit 2024-04-24T17:27:37.265004+0000 mon.a (mon.0) 53 : audit [DBG] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T17:27:38.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:38 smithi032 bash[21457]: audit 2024-04-24T17:27:37.265004+0000 mon.a (mon.0) 53 : audit [DBG] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T17:27:38.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:38 smithi032 bash[21457]: audit 2024-04-24T17:27:37.274313+0000 mon.a (mon.0) 54 : audit [INF] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T17:27:38.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:38 smithi032 bash[21457]: audit 2024-04-24T17:27:37.274313+0000 mon.a (mon.0) 54 : audit [INF] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T17:27:38.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:38 smithi032 bash[21457]: audit 2024-04-24T17:27:37.277318+0000 mon.a (mon.0) 55 : audit [INF] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T17:27:38.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:38 smithi032 bash[21457]: audit 2024-04-24T17:27:37.277318+0000 mon.a (mon.0) 55 : audit [INF] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T17:27:38.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:38 smithi032 bash[21457]: audit 2024-04-24T17:27:37.542101+0000 mon.a (mon.0) 56 : audit [INF] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' 2024-04-24T17:27:38.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:38 smithi032 bash[21457]: audit 2024-04-24T17:27:37.542101+0000 mon.a (mon.0) 56 : audit [INF] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' 2024-04-24T17:27:38.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:38 smithi032 bash[21457]: audit 2024-04-24T17:27:37.545422+0000 mon.a (mon.0) 57 : audit [INF] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' 2024-04-24T17:27:38.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:38 smithi032 bash[21457]: audit 2024-04-24T17:27:37.545422+0000 mon.a (mon.0) 57 : audit [INF] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' 2024-04-24T17:27:38.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:38 smithi032 bash[21457]: cluster 2024-04-24T17:27:38.202717+0000 mon.a (mon.0) 58 : cluster [DBG] mgrmap e7: y(active, since 1.01044s) 2024-04-24T17:27:38.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:38 smithi032 bash[21457]: cluster 2024-04-24T17:27:38.202717+0000 mon.a (mon.0) 58 : cluster [DBG] mgrmap e7: y(active, since 1.01044s) 2024-04-24T17:27:38.905 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-04-24T17:27:38.905 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-24T17:27:38.905 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-24T17:27:38.905 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-04-24T17:27:38.905 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.309+0000 7f389c24a700 1 Processor -- start 2024-04-24T17:27:38.905 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.309+0000 7f389c24a700 1 -- start start 2024-04-24T17:27:38.905 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.309+0000 7f389c24a700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f389410deb0 con 0x7f3894109a80 2024-04-24T17:27:38.905 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.309+0000 7f3899fe6700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3894109a80 0x7f3894109ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55296/0 (socket says 172.21.15.32:55296) 2024-04-24T17:27:38.905 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.309+0000 7f3899fe6700 1 -- 172.21.15.32:0/702813618 learned_addr learned my addr 172.21.15.32:0/702813618 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:27:38.905 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.309+0000 7f3898fe4700 1 -- 172.21.15.32:0/702813618 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4072628997 0 0) 0x7f389410deb0 con 0x7f3894109a80 2024-04-24T17:27:38.906 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.309+0000 7f3898fe4700 1 -- 172.21.15.32:0/702813618 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f38800035b0 con 0x7f3894109a80 2024-04-24T17:27:38.906 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.309+0000 7f3898fe4700 1 -- 172.21.15.32:0/702813618 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1087630005 0 0) 0x7f38800035b0 con 0x7f3894109a80 2024-04-24T17:27:38.906 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.309+0000 7f3898fe4700 1 -- 172.21.15.32:0/702813618 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f389410f090 con 0x7f3894109a80 2024-04-24T17:27:38.906 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.309+0000 7f3898fe4700 1 -- 172.21.15.32:0/702813618 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f3884001580 con 0x7f3894109a80 2024-04-24T17:27:38.906 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.309+0000 7f3898fe4700 1 -- 172.21.15.32:0/702813618 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(25 keys) ==== 1011+0+0 (unknown 2148541123 0 0) 0x7f3884001d10 con 0x7f3894109a80 2024-04-24T17:27:38.906 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.309+0000 7f3898fe4700 1 -- 172.21.15.32:0/702813618 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f3884005960 con 0x7f3894109a80 2024-04-24T17:27:38.906 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.309+0000 7f389c24a700 1 -- 172.21.15.32:0/702813618 >> v1:172.21.15.32:6789/0 conn(0x7f3894109a80 legacy=0x7f3894109ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:38.906 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.309+0000 7f389c24a700 1 -- 172.21.15.32:0/702813618 shutdown_connections 2024-04-24T17:27:38.906 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.309+0000 7f389c24a700 1 -- 172.21.15.32:0/702813618 >> 172.21.15.32:0/702813618 conn(0x7f389407ce10 msgr2=0x7f38941074a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:27:38.906 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.309+0000 7f389c24a700 1 -- 172.21.15.32:0/702813618 shutdown_connections 2024-04-24T17:27:38.906 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.309+0000 7f389c24a700 1 -- 172.21.15.32:0/702813618 wait complete. 2024-04-24T17:27:38.906 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.313+0000 7f389c24a700 1 librados: starting msgr at 2024-04-24T17:27:38.906 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.313+0000 7f389c24a700 1 librados: starting objecter 2024-04-24T17:27:38.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.313+0000 7f389c24a700 1 Processor -- start 2024-04-24T17:27:38.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.313+0000 7f389c24a700 1 -- start start 2024-04-24T17:27:38.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.313+0000 7f389c24a700 1 librados: setting wanted keys 2024-04-24T17:27:38.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.313+0000 7f389c24a700 1 librados: calling monclient init 2024-04-24T17:27:38.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.313+0000 7f389c24a700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f389419dda0 con 0x7f3894109a80 2024-04-24T17:27:38.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.313+0000 7f3899fe6700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3894109a80 0x7f389419d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:55306/0 (socket says 172.21.15.32:55306) 2024-04-24T17:27:38.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.313+0000 7f3899fe6700 1 -- 172.21.15.32:0/3461694543 learned_addr learned my addr 172.21.15.32:0/3461694543 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:27:38.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.313+0000 7f388affd700 10 client.?.objecter ms_handle_connect 0x7f3894109a80 2024-04-24T17:27:38.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.313+0000 7f388affd700 10 client.?.objecter resend_mon_ops 2024-04-24T17:27:38.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.313+0000 7f388affd700 1 -- 172.21.15.32:0/3461694543 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2846330590 0 0) 0x7f389419dda0 con 0x7f3894109a80 2024-04-24T17:27:38.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.313+0000 7f388affd700 1 -- 172.21.15.32:0/3461694543 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f38800056e0 con 0x7f3894109a80 2024-04-24T17:27:38.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.313+0000 7f388affd700 1 -- 172.21.15.32:0/3461694543 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1041120611 0 0) 0x7f389410f090 con 0x7f3894109a80 2024-04-24T17:27:38.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.313+0000 7f388affd700 1 -- 172.21.15.32:0/3461694543 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f389419dda0 con 0x7f3894109a80 2024-04-24T17:27:38.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.313+0000 7f388affd700 1 -- 172.21.15.32:0/3461694543 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f3884001080 con 0x7f3894109a80 2024-04-24T17:27:38.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.313+0000 7f388affd700 1 -- 172.21.15.32:0/3461694543 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3064805193 0 0) 0x7f389419dda0 con 0x7f3894109a80 2024-04-24T17:27:38.908 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.313+0000 7f388affd700 1 -- 172.21.15.32:0/3461694543 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f389419df70 con 0x7f3894109a80 2024-04-24T17:27:38.908 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.313+0000 7f389c24a700 1 -- 172.21.15.32:0/3461694543 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f389419e260 con 0x7f3894109a80 2024-04-24T17:27:38.908 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.313+0000 7f389c24a700 10 client.14124.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-24T17:27:38.908 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.313+0000 7f389c24a700 1 -- 172.21.15.32:0/3461694543 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f38941a1db0 con 0x7f3894109a80 2024-04-24T17:27:38.908 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.313+0000 7f389c24a700 1 librados: init done 2024-04-24T17:27:38.908 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.313+0000 7f388affd700 1 -- 172.21.15.32:0/3461694543 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(25 keys) ==== 1011+0+0 (unknown 2148541123 0 0) 0x7f3884001d10 con 0x7f3894109a80 2024-04-24T17:27:38.908 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.313+0000 7f388affd700 1 -- 172.21.15.32:0/3461694543 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f3884006020 con 0x7f3894109a80 2024-04-24T17:27:38.908 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.313+0000 7f388affd700 1 -- 172.21.15.32:0/3461694543 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 5) ==== 46523+0+0 (unknown 235251547 0 0) 0x7f38840061c0 con 0x7f3894109a80 2024-04-24T17:27:38.908 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.313+0000 7f388affd700 1 -- 172.21.15.32:0/3461694543 --> v1:172.21.15.32:6800/3443430034 -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f3884001080 con 0x7f388003c310 2024-04-24T17:27:38.908 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.313+0000 7f388affd700 1 -- 172.21.15.32:0/3461694543 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 2428325519 0 0) 0x7f3884048d60 con 0x7f3894109a80 2024-04-24T17:27:38.908 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.313+0000 7f388affd700 10 client.14124.objecter ms_dispatch 0x7f3894001550 osd_map(1..1 src has 1..1) 2024-04-24T17:27:38.908 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.313+0000 7f388affd700 3 client.14124.objecter handle_osd_map got epochs [1,1] > 0 2024-04-24T17:27:38.908 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.313+0000 7f388affd700 3 client.14124.objecter handle_osd_map decoding full epoch 1 2024-04-24T17:27:38.909 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.313+0000 7f38997e5700 1 -- 172.21.15.32:0/3461694543 >> v1:172.21.15.32:6800/3443430034 conn(0x7f388003c310 legacy=0x7f388003e7d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.32:6800/3443430034 2024-04-24T17:27:38.909 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.313+0000 7f388affd700 20 client.14124.objecter dump_active .. 0 homeless 2024-04-24T17:27:38.909 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.517+0000 7f38997e5700 1 -- 172.21.15.32:0/3461694543 >> v1:172.21.15.32:6800/3443430034 conn(0x7f388003c310 legacy=0x7f388003e7d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.32:6800/3443430034 2024-04-24T17:27:38.909 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:32.921+0000 7f38997e5700 1 -- 172.21.15.32:0/3461694543 >> v1:172.21.15.32:6800/3443430034 conn(0x7f388003c310 legacy=0x7f388003e7d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.32:6800/3443430034 2024-04-24T17:27:38.909 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:33.725+0000 7f38997e5700 1 -- 172.21.15.32:0/3461694543 >> v1:172.21.15.32:6800/3443430034 conn(0x7f388003c310 legacy=0x7f388003e7d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.32:6800/3443430034 2024-04-24T17:27:38.909 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:35.329+0000 7f38997e5700 1 -- 172.21.15.32:0/3461694543 >> v1:172.21.15.32:6800/3443430034 conn(0x7f388003c310 legacy=0x7f388003e7d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.32:6800/3443430034 2024-04-24T17:27:38.909 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:37.193+0000 7f388affd700 1 -- 172.21.15.32:0/3461694543 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mgrmap(e 6) ==== 46325+0+0 (unknown 3372169964 0 0) 0x7f3884004a40 con 0x7f3894109a80 2024-04-24T17:27:38.909 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:37.193+0000 7f388affd700 1 -- 172.21.15.32:0/3461694543 >> v1:172.21.15.32:6800/3443430034 conn(0x7f388003c310 legacy=0x7f388003e7d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T17:27:38.909 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:37.317+0000 7f389ba49700 10 client.14124.objecter tick 2024-04-24T17:27:38.909 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:38.197+0000 7f388affd700 1 -- 172.21.15.32:0/3461694543 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mgrmap(e 7) ==== 46417+0+0 (unknown 2101159598 0 0) 0x7f3884049290 con 0x7f3894109a80 2024-04-24T17:27:38.909 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:38.197+0000 7f388affd700 1 -- 172.21.15.32:0/3461694543 --> v1:172.21.15.32:6800/3410850029 -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f388401b2b0 con 0x7f3880040bd0 2024-04-24T17:27:38.909 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:38.197+0000 7f388affd700 10 client.14124.objecter ms_handle_connect 0x7f3880040bd0 2024-04-24T17:27:38.909 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:38.197+0000 7f388affd700 1 -- 172.21.15.32:0/3461694543 <== mgr.14120 v1:172.21.15.32:6800/3410850029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (unknown 0 0 1361822222) 0x7f388401b2b0 con 0x7f3880040bd0 2024-04-24T17:27:38.909 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:38.213+0000 7f389c24a700 1 -- 172.21.15.32:0/3461694543 --> v1:172.21.15.32:6800/3410850029 -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f3878002630 con 0x7f3880040bd0 2024-04-24T17:27:38.909 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:38.213+0000 7f388affd700 1 -- 172.21.15.32:0/3461694543 <== mgr.14120 v1:172.21.15.32:6800/3410850029 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (unknown 0 0 96372106) 0x7f3878002630 con 0x7f3880040bd0 2024-04-24T17:27:38.910 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:38.213+0000 7f389c24a700 10 librados: watch_flush enter 2024-04-24T17:27:38.910 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:38.213+0000 7f389c24a700 10 librados: watch_flush exit 2024-04-24T17:27:38.910 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:38.213+0000 7f389c24a700 20 client.14124.objecter shutdown clearing up homeless session... 2024-04-24T17:27:38.910 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:38.213+0000 7f389c24a700 10 client.14124.objecter successfully canceled tick 2024-04-24T17:27:38.910 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:38.213+0000 7f389c24a700 1 -- 172.21.15.32:0/3461694543 >> v1:172.21.15.32:6800/3410850029 conn(0x7f3880040bd0 legacy=0x7f3880042fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:38.910 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:38.213+0000 7f389c24a700 1 -- 172.21.15.32:0/3461694543 >> v1:172.21.15.32:6789/0 conn(0x7f3894109a80 legacy=0x7f389419d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:38.910 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:38.213+0000 7f389c24a700 1 -- 172.21.15.32:0/3461694543 shutdown_connections 2024-04-24T17:27:38.910 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:38.213+0000 7f389c24a700 1 -- 172.21.15.32:0/3461694543 >> 172.21.15.32:0/3461694543 conn(0x7f389407ce10 msgr2=0x7f38941074a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:27:38.910 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:38.213+0000 7f389c24a700 1 -- 172.21.15.32:0/3461694543 shutdown_connections 2024-04-24T17:27:38.910 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:38.213+0000 7f389c24a700 1 -- 172.21.15.32:0/3461694543 wait complete. 2024-04-24T17:27:38.910 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:38.213+0000 7f389c24a700 1 librados: shutdown 2024-04-24T17:27:38.910 INFO:teuthology.orchestra.run.smithi032.stdout:mgr epoch 5 is available 2024-04-24T17:27:38.910 INFO:teuthology.orchestra.run.smithi032.stdout:Setting orchestrator backend to cephadm... 2024-04-24T17:27:39.599 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:39 smithi032 bash[21457]: audit 2024-04-24T17:27:38.201124+0000 mgr.y (mgr.14120) 2 : audit [DBG] from='client.14124 v1:172.21.15.32:0/3461694543' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-24T17:27:39.599 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:39 smithi032 bash[21457]: audit 2024-04-24T17:27:38.201124+0000 mgr.y (mgr.14120) 2 : audit [DBG] from='client.14124 v1:172.21.15.32:0/3461694543' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-24T17:27:39.599 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:39 smithi032 bash[21457]: audit 2024-04-24T17:27:38.216733+0000 mgr.y (mgr.14120) 3 : audit [DBG] from='client.14124 v1:172.21.15.32:0/3461694543' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-24T17:27:39.599 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:39 smithi032 bash[21457]: audit 2024-04-24T17:27:38.216733+0000 mgr.y (mgr.14120) 3 : audit [DBG] from='client.14124 v1:172.21.15.32:0/3461694543' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-24T17:27:39.599 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:39 smithi032 bash[21457]: cephadm 2024-04-24T17:27:38.221748+0000 mgr.y (mgr.14120) 4 : cephadm [INF] [24/Apr/2024:17:27:38] ENGINE Bus STARTING 2024-04-24T17:27:39.599 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:39 smithi032 bash[21457]: cephadm 2024-04-24T17:27:38.221748+0000 mgr.y (mgr.14120) 4 : cephadm [INF] [24/Apr/2024:17:27:38] ENGINE Bus STARTING 2024-04-24T17:27:39.599 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:39 smithi032 bash[21457]: cephadm 2024-04-24T17:27:38.343857+0000 mgr.y (mgr.14120) 5 : cephadm [INF] [24/Apr/2024:17:27:38] ENGINE Serving on https://172.21.15.32:7150 2024-04-24T17:27:39.599 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:39 smithi032 bash[21457]: cephadm 2024-04-24T17:27:38.343857+0000 mgr.y (mgr.14120) 5 : cephadm [INF] [24/Apr/2024:17:27:38] ENGINE Serving on https://172.21.15.32:7150 2024-04-24T17:27:39.599 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:39 smithi032 bash[21457]: cephadm 2024-04-24T17:27:38.445438+0000 mgr.y (mgr.14120) 6 : cephadm [INF] [24/Apr/2024:17:27:38] ENGINE Serving on http://172.21.15.32:8765 2024-04-24T17:27:39.599 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:39 smithi032 bash[21457]: cephadm 2024-04-24T17:27:38.445438+0000 mgr.y (mgr.14120) 6 : cephadm [INF] [24/Apr/2024:17:27:38] ENGINE Serving on http://172.21.15.32:8765 2024-04-24T17:27:39.599 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:39 smithi032 bash[21457]: cephadm 2024-04-24T17:27:38.446011+0000 mgr.y (mgr.14120) 7 : cephadm [INF] [24/Apr/2024:17:27:38] ENGINE Bus STARTED 2024-04-24T17:27:39.599 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:39 smithi032 bash[21457]: cephadm 2024-04-24T17:27:38.446011+0000 mgr.y (mgr.14120) 7 : cephadm [INF] [24/Apr/2024:17:27:38] ENGINE Bus STARTED 2024-04-24T17:27:39.599 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:39 smithi032 bash[21457]: audit 2024-04-24T17:27:38.484886+0000 mon.a (mon.0) 59 : audit [DBG] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T17:27:39.600 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:39 smithi032 bash[21457]: audit 2024-04-24T17:27:38.484886+0000 mon.a (mon.0) 59 : audit [DBG] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T17:27:40.751 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:40 smithi032 bash[21457]: cluster 2024-04-24T17:27:39.267658+0000 mon.a (mon.0) 60 : cluster [DBG] mgrmap e8: y(active, since 2s) 2024-04-24T17:27:40.751 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:40 smithi032 bash[21457]: cluster 2024-04-24T17:27:39.267658+0000 mon.a (mon.0) 60 : cluster [DBG] mgrmap e8: y(active, since 2s) 2024-04-24T17:27:40.751 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:40 smithi032 bash[21457]: audit 2024-04-24T17:27:40.249320+0000 mon.a (mon.0) 61 : audit [INF] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' 2024-04-24T17:27:40.752 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:40 smithi032 bash[21457]: audit 2024-04-24T17:27:40.249320+0000 mon.a (mon.0) 61 : audit [INF] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' 2024-04-24T17:27:40.850 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.809+0000 7f20e72e1700 1 Processor -- start 2024-04-24T17:27:40.850 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.809+0000 7f20e72e1700 1 -- start start 2024-04-24T17:27:40.850 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.809+0000 7f20e72e1700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f20e010ded0 con 0x7f20e0109aa0 2024-04-24T17:27:40.850 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.809+0000 7f20e507d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f20e0109aa0 0x7f20e0109ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:46678/0 (socket says 172.21.15.32:46678) 2024-04-24T17:27:40.851 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.809+0000 7f20e507d700 1 -- 172.21.15.32:0/2933424950 learned_addr learned my addr 172.21.15.32:0/2933424950 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:27:40.851 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.809+0000 7f20d7fff700 1 -- 172.21.15.32:0/2933424950 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1001331928 0 0) 0x7f20e010ded0 con 0x7f20e0109aa0 2024-04-24T17:27:40.851 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.809+0000 7f20d7fff700 1 -- 172.21.15.32:0/2933424950 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20cc0035b0 con 0x7f20e0109aa0 2024-04-24T17:27:40.851 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.809+0000 7f20d7fff700 1 -- 172.21.15.32:0/2933424950 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2847358111 0 0) 0x7f20cc0035b0 con 0x7f20e0109aa0 2024-04-24T17:27:40.851 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.809+0000 7f20d7fff700 1 -- 172.21.15.32:0/2933424950 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20e010f0b0 con 0x7f20e0109aa0 2024-04-24T17:27:40.851 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.809+0000 7f20d7fff700 1 -- 172.21.15.32:0/2933424950 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f20dc001580 con 0x7f20e0109aa0 2024-04-24T17:27:40.851 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.809+0000 7f20d7fff700 1 -- 172.21.15.32:0/2933424950 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(25 keys) ==== 1011+0+0 (unknown 2148541123 0 0) 0x7f20dc001c40 con 0x7f20e0109aa0 2024-04-24T17:27:40.851 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.809+0000 7f20d7fff700 1 -- 172.21.15.32:0/2933424950 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f20dc005750 con 0x7f20e0109aa0 2024-04-24T17:27:40.851 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.809+0000 7f20e72e1700 1 -- 172.21.15.32:0/2933424950 >> v1:172.21.15.32:6789/0 conn(0x7f20e0109aa0 legacy=0x7f20e0109ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:40.851 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.809+0000 7f20e72e1700 1 -- 172.21.15.32:0/2933424950 shutdown_connections 2024-04-24T17:27:40.851 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.809+0000 7f20e72e1700 1 -- 172.21.15.32:0/2933424950 >> 172.21.15.32:0/2933424950 conn(0x7f20e007ced0 msgr2=0x7f20e01074a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:27:40.851 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.809+0000 7f20e72e1700 1 -- 172.21.15.32:0/2933424950 shutdown_connections 2024-04-24T17:27:40.851 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.809+0000 7f20e72e1700 1 -- 172.21.15.32:0/2933424950 wait complete. 2024-04-24T17:27:40.851 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.809+0000 7f20e72e1700 1 librados: starting msgr at 2024-04-24T17:27:40.852 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.809+0000 7f20e72e1700 1 librados: starting objecter 2024-04-24T17:27:40.852 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.809+0000 7f20e72e1700 1 Processor -- start 2024-04-24T17:27:40.852 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.809+0000 7f20e72e1700 1 -- start start 2024-04-24T17:27:40.852 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.809+0000 7f20e72e1700 1 librados: setting wanted keys 2024-04-24T17:27:40.852 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.809+0000 7f20e72e1700 1 librados: calling monclient init 2024-04-24T17:27:40.852 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.809+0000 7f20e72e1700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f20e019e0b0 con 0x7f20e019d580 2024-04-24T17:27:40.852 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.809+0000 7f20e507d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f20e019d580 0x7f20e019d9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:46690/0 (socket says 172.21.15.32:46690) 2024-04-24T17:27:40.852 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.809+0000 7f20e507d700 1 -- 172.21.15.32:0/1935365360 learned_addr learned my addr 172.21.15.32:0/1935365360 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:27:40.852 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.809+0000 7f20d67fc700 10 client.?.objecter ms_handle_connect 0x7f20e019d580 2024-04-24T17:27:40.852 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.809+0000 7f20d67fc700 10 client.?.objecter resend_mon_ops 2024-04-24T17:27:40.852 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.813+0000 7f20d67fc700 1 -- 172.21.15.32:0/1935365360 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2067317050 0 0) 0x7f20e019e0b0 con 0x7f20e019d580 2024-04-24T17:27:40.852 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.813+0000 7f20d67fc700 1 -- 172.21.15.32:0/1935365360 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20cc0035b0 con 0x7f20e019d580 2024-04-24T17:27:40.852 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.813+0000 7f20d67fc700 1 -- 172.21.15.32:0/1935365360 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1051210416 0 0) 0x7f20cc0035b0 con 0x7f20e019d580 2024-04-24T17:27:40.852 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.813+0000 7f20d67fc700 1 -- 172.21.15.32:0/1935365360 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f20e019e0b0 con 0x7f20e019d580 2024-04-24T17:27:40.852 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.813+0000 7f20d67fc700 1 -- 172.21.15.32:0/1935365360 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f20dc001080 con 0x7f20e019d580 2024-04-24T17:27:40.853 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.813+0000 7f20d67fc700 1 -- 172.21.15.32:0/1935365360 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2614733013 0 0) 0x7f20e019e0b0 con 0x7f20e019d580 2024-04-24T17:27:40.853 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.813+0000 7f20d67fc700 1 -- 172.21.15.32:0/1935365360 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20e01a2aa0 con 0x7f20e019d580 2024-04-24T17:27:40.853 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.813+0000 7f20e72e1700 1 -- 172.21.15.32:0/1935365360 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f20e01a1a90 con 0x7f20e019d580 2024-04-24T17:27:40.853 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.813+0000 7f20e72e1700 10 client.14132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-24T17:27:40.853 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.813+0000 7f20e72e1700 1 -- 172.21.15.32:0/1935365360 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f20e01a1fd0 con 0x7f20e019d580 2024-04-24T17:27:40.853 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.813+0000 7f20e72e1700 1 librados: init done 2024-04-24T17:27:40.853 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.813+0000 7f20d67fc700 1 -- 172.21.15.32:0/1935365360 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(25 keys) ==== 1011+0+0 (unknown 2148541123 0 0) 0x7f20dc001810 con 0x7f20e019d580 2024-04-24T17:27:40.853 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.813+0000 7f20d67fc700 1 -- 172.21.15.32:0/1935365360 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f20dc005f80 con 0x7f20e019d580 2024-04-24T17:27:40.853 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.813+0000 7f20d67fc700 1 -- 172.21.15.32:0/1935365360 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 8) ==== 46523+0+0 (unknown 1814385894 0 0) 0x7f20dc0116f0 con 0x7f20e019d580 2024-04-24T17:27:40.853 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.813+0000 7f20e72e1700 1 -- 172.21.15.32:0/1935365360 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20c4005150 con 0x7f20e019d580 2024-04-24T17:27:40.853 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.813+0000 7f20d67fc700 1 -- 172.21.15.32:0/1935365360 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 3338368936 0 0) 0x7f20dc048330 con 0x7f20e019d580 2024-04-24T17:27:40.853 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.813+0000 7f20d67fc700 10 client.14132.objecter ms_dispatch 0x7f20e0001550 osd_map(2..2 src has 1..2) 2024-04-24T17:27:40.853 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.813+0000 7f20d67fc700 3 client.14132.objecter handle_osd_map got epochs [2,2] > 0 2024-04-24T17:27:40.853 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.813+0000 7f20d67fc700 3 client.14132.objecter handle_osd_map decoding full epoch 2 2024-04-24T17:27:40.853 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.813+0000 7f20d67fc700 20 client.14132.objecter dump_active .. 0 homeless 2024-04-24T17:27:40.854 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.813+0000 7f20d67fc700 10 client.14132.objecter ms_handle_connect 0x7f20cc03c3c0 2024-04-24T17:27:40.854 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:39.817+0000 7f20d67fc700 1 -- 172.21.15.32:0/1935365360 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188728 (unknown 1092875540 0 2656817166) 0x7f20dc017620 con 0x7f20e019d580 2024-04-24T17:27:40.854 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:40.237+0000 7f20e72e1700 1 -- 172.21.15.32:0/1935365360 --> v1:172.21.15.32:6800/3410850029 -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f20c4000bc0 con 0x7f20cc03c3c0 2024-04-24T17:27:40.854 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:40.245+0000 7f20d67fc700 1 -- 172.21.15.32:0/1935365360 <== mgr.14120 v1:172.21.15.32:6800/3410850029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (unknown 0 0 0) 0x7f20c4000bc0 con 0x7f20cc03c3c0 2024-04-24T17:27:40.854 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:40.249+0000 7f20e72e1700 10 librados: watch_flush enter 2024-04-24T17:27:40.854 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:40.249+0000 7f20e72e1700 10 librados: watch_flush exit 2024-04-24T17:27:40.854 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:40.249+0000 7f20e72e1700 20 client.14132.objecter shutdown clearing up homeless session... 2024-04-24T17:27:40.854 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:40.249+0000 7f20e72e1700 10 client.14132.objecter successfully canceled tick 2024-04-24T17:27:40.854 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:40.249+0000 7f20e72e1700 1 -- 172.21.15.32:0/1935365360 >> v1:172.21.15.32:6800/3410850029 conn(0x7f20cc03c3c0 legacy=0x7f20cc03e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:40.854 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:40.249+0000 7f20e72e1700 1 -- 172.21.15.32:0/1935365360 >> v1:172.21.15.32:6789/0 conn(0x7f20e019d580 legacy=0x7f20e019d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:40.854 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:40.249+0000 7f20e72e1700 1 -- 172.21.15.32:0/1935365360 shutdown_connections 2024-04-24T17:27:40.854 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:40.249+0000 7f20e72e1700 1 -- 172.21.15.32:0/1935365360 >> 172.21.15.32:0/1935365360 conn(0x7f20e0109aa0 msgr2=0x7f20e01074a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:27:40.854 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:40.249+0000 7f20e72e1700 1 -- 172.21.15.32:0/1935365360 shutdown_connections 2024-04-24T17:27:40.854 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:40.249+0000 7f20e72e1700 1 -- 172.21.15.32:0/1935365360 wait complete. 2024-04-24T17:27:40.854 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:40.249+0000 7f20e72e1700 1 librados: shutdown 2024-04-24T17:27:41.517 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:41 smithi032 bash[21457]: audit 2024-04-24T17:27:40.241115+0000 mgr.y (mgr.14120) 8 : audit [DBG] from='client.14132 v1:172.21.15.32:0/1935365360' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T17:27:41.517 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:41 smithi032 bash[21457]: audit 2024-04-24T17:27:40.241115+0000 mgr.y (mgr.14120) 8 : audit [DBG] from='client.14132 v1:172.21.15.32:0/1935365360' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T17:27:41.518 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:41 smithi032 bash[21457]: audit 2024-04-24T17:27:40.306333+0000 mon.a (mon.0) 62 : audit [DBG] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T17:27:41.518 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:41 smithi032 bash[21457]: audit 2024-04-24T17:27:40.306333+0000 mon.a (mon.0) 62 : audit [DBG] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T17:27:42.820 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-24T17:27:42.821 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.701+0000 7f4ef6535700 1 Processor -- start 2024-04-24T17:27:42.821 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.701+0000 7f4ef6535700 1 -- start start 2024-04-24T17:27:42.821 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.701+0000 7f4ef6535700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4ef010ded0 con 0x7f4ef0109aa0 2024-04-24T17:27:42.821 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.701+0000 7f4eeffff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f4ef0109aa0 0x7f4ef0109ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:46698/0 (socket says 172.21.15.32:46698) 2024-04-24T17:27:42.821 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.701+0000 7f4eeffff700 1 -- 172.21.15.32:0/2219520972 learned_addr learned my addr 172.21.15.32:0/2219520972 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:27:42.821 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.701+0000 7f4eeeffd700 1 -- 172.21.15.32:0/2219520972 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2663120577 0 0) 0x7f4ef010ded0 con 0x7f4ef0109aa0 2024-04-24T17:27:42.821 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.701+0000 7f4eeeffd700 1 -- 172.21.15.32:0/2219520972 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ed80035b0 con 0x7f4ef0109aa0 2024-04-24T17:27:42.821 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.701+0000 7f4eeeffd700 1 -- 172.21.15.32:0/2219520972 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2058435908 0 0) 0x7f4ed80035b0 con 0x7f4ef0109aa0 2024-04-24T17:27:42.821 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.701+0000 7f4eeeffd700 1 -- 172.21.15.32:0/2219520972 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ef010f0b0 con 0x7f4ef0109aa0 2024-04-24T17:27:42.821 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.701+0000 7f4eeeffd700 1 -- 172.21.15.32:0/2219520972 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f4ee0001900 con 0x7f4ef0109aa0 2024-04-24T17:27:42.821 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.701+0000 7f4eeeffd700 1 -- 172.21.15.32:0/2219520972 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(25 keys) ==== 1011+0+0 (unknown 2148541123 0 0) 0x7f4ee0002090 con 0x7f4ef0109aa0 2024-04-24T17:27:42.821 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.701+0000 7f4eeeffd700 1 -- 172.21.15.32:0/2219520972 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f4ee0005ce0 con 0x7f4ef0109aa0 2024-04-24T17:27:42.821 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.701+0000 7f4ef6535700 1 -- 172.21.15.32:0/2219520972 >> v1:172.21.15.32:6789/0 conn(0x7f4ef0109aa0 legacy=0x7f4ef0109ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:42.822 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.701+0000 7f4ef6535700 1 -- 172.21.15.32:0/2219520972 shutdown_connections 2024-04-24T17:27:42.822 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.701+0000 7f4ef6535700 1 -- 172.21.15.32:0/2219520972 >> 172.21.15.32:0/2219520972 conn(0x7f4ef007ceb0 msgr2=0x7f4ef01074a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:27:42.822 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.701+0000 7f4ef6535700 1 -- 172.21.15.32:0/2219520972 shutdown_connections 2024-04-24T17:27:42.822 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.705+0000 7f4ef6535700 1 -- 172.21.15.32:0/2219520972 wait complete. 2024-04-24T17:27:42.822 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.705+0000 7f4ef6535700 1 librados: starting msgr at 2024-04-24T17:27:42.822 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.705+0000 7f4ef6535700 1 librados: starting objecter 2024-04-24T17:27:42.822 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.705+0000 7f4ef6535700 1 Processor -- start 2024-04-24T17:27:42.822 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.705+0000 7f4ef6535700 1 -- start start 2024-04-24T17:27:42.822 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.705+0000 7f4ef6535700 1 librados: setting wanted keys 2024-04-24T17:27:42.822 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.705+0000 7f4ef6535700 1 librados: calling monclient init 2024-04-24T17:27:42.822 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.705+0000 7f4ef6535700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4ef019dd30 con 0x7f4ef0109aa0 2024-04-24T17:27:42.822 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.705+0000 7f4eeffff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f4ef0109aa0 0x7f4ef019d620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:46708/0 (socket says 172.21.15.32:46708) 2024-04-24T17:27:42.826 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.705+0000 7f4eeffff700 1 -- 172.21.15.32:0/4031806217 learned_addr learned my addr 172.21.15.32:0/4031806217 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:27:42.826 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.705+0000 7f4eed7fa700 10 client.?.objecter ms_handle_connect 0x7f4ef0109aa0 2024-04-24T17:27:42.826 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.705+0000 7f4eed7fa700 10 client.?.objecter resend_mon_ops 2024-04-24T17:27:42.827 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.705+0000 7f4eed7fa700 1 -- 172.21.15.32:0/4031806217 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1436868338 0 0) 0x7f4ef019dd30 con 0x7f4ef0109aa0 2024-04-24T17:27:42.827 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.705+0000 7f4eed7fa700 1 -- 172.21.15.32:0/4031806217 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ed80056e0 con 0x7f4ef0109aa0 2024-04-24T17:27:42.827 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.705+0000 7f4eed7fa700 1 -- 172.21.15.32:0/4031806217 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 51090738 0 0) 0x7f4ef010f0b0 con 0x7f4ef0109aa0 2024-04-24T17:27:42.827 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.705+0000 7f4eed7fa700 1 -- 172.21.15.32:0/4031806217 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4ef019dd30 con 0x7f4ef0109aa0 2024-04-24T17:27:42.827 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.705+0000 7f4eed7fa700 1 -- 172.21.15.32:0/4031806217 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f4ee0004bf0 con 0x7f4ef0109aa0 2024-04-24T17:27:42.827 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.705+0000 7f4eed7fa700 1 -- 172.21.15.32:0/4031806217 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4116934958 0 0) 0x7f4ef019dd30 con 0x7f4ef0109aa0 2024-04-24T17:27:42.827 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.705+0000 7f4eed7fa700 1 -- 172.21.15.32:0/4031806217 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ef019df00 con 0x7f4ef0109aa0 2024-04-24T17:27:42.827 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.705+0000 7f4ef6535700 1 -- 172.21.15.32:0/4031806217 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4ef019e1f0 con 0x7f4ef0109aa0 2024-04-24T17:27:42.827 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.705+0000 7f4ef6535700 10 client.14134.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-24T17:27:42.827 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.705+0000 7f4ef6535700 1 -- 172.21.15.32:0/4031806217 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4ef01a1d40 con 0x7f4ef0109aa0 2024-04-24T17:27:42.827 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.705+0000 7f4ef6535700 1 librados: init done 2024-04-24T17:27:42.827 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.705+0000 7f4eed7fa700 1 -- 172.21.15.32:0/4031806217 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(25 keys) ==== 1011+0+0 (unknown 2148541123 0 0) 0x7f4ee0002090 con 0x7f4ef0109aa0 2024-04-24T17:27:42.827 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.705+0000 7f4eed7fa700 1 -- 172.21.15.32:0/4031806217 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f4ee0006340 con 0x7f4ef0109aa0 2024-04-24T17:27:42.827 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.705+0000 7f4eed7fa700 1 -- 172.21.15.32:0/4031806217 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 8) ==== 46523+0+0 (unknown 1814385894 0 0) 0x7f4ee0011ab0 con 0x7f4ef0109aa0 2024-04-24T17:27:42.828 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.705+0000 7f4ef6535700 1 -- 172.21.15.32:0/4031806217 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4edc005150 con 0x7f4ef0109aa0 2024-04-24T17:27:42.828 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.705+0000 7f4eed7fa700 1 -- 172.21.15.32:0/4031806217 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 3338368936 0 0) 0x7f4ee0048310 con 0x7f4ef0109aa0 2024-04-24T17:27:42.828 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.705+0000 7f4eed7fa700 10 client.14134.objecter ms_dispatch 0x7f4ef0001550 osd_map(2..2 src has 1..2) 2024-04-24T17:27:42.828 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.705+0000 7f4eed7fa700 3 client.14134.objecter handle_osd_map got epochs [2,2] > 0 2024-04-24T17:27:42.828 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.705+0000 7f4eed7fa700 3 client.14134.objecter handle_osd_map decoding full epoch 2 2024-04-24T17:27:42.828 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.705+0000 7f4eed7fa700 20 client.14134.objecter dump_active .. 0 homeless 2024-04-24T17:27:42.828 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.705+0000 7f4eed7fa700 10 client.14134.objecter ms_handle_connect 0x7f4ed803c220 2024-04-24T17:27:42.828 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:41.713+0000 7f4eed7fa700 1 -- 172.21.15.32:0/4031806217 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188728 (unknown 1092875540 0 2656817166) 0x7f4ed80056e0 con 0x7f4ef0109aa0 2024-04-24T17:27:42.828 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:42.101+0000 7f4ef6535700 1 -- 172.21.15.32:0/4031806217 --> v1:172.21.15.32:6800/3410850029 -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f4edc000bc0 con 0x7f4ed803c220 2024-04-24T17:27:42.828 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:42.105+0000 7f4eed7fa700 1 -- 172.21.15.32:0/4031806217 <== mgr.14120 v1:172.21.15.32:6800/3410850029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (unknown 0 0 2070689548) 0x7f4edc000bc0 con 0x7f4ed803c220 2024-04-24T17:27:42.828 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:42.105+0000 7f4ef6535700 10 librados: watch_flush enter 2024-04-24T17:27:42.828 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:42.105+0000 7f4ef6535700 10 librados: watch_flush exit 2024-04-24T17:27:42.828 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:42.105+0000 7f4ef6535700 20 client.14134.objecter shutdown clearing up homeless session... 2024-04-24T17:27:42.828 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:42.105+0000 7f4ef6535700 10 client.14134.objecter successfully canceled tick 2024-04-24T17:27:42.828 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:42.105+0000 7f4ef6535700 1 -- 172.21.15.32:0/4031806217 >> v1:172.21.15.32:6800/3410850029 conn(0x7f4ed803c220 legacy=0x7f4ed803e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:42.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:42.105+0000 7f4ef6535700 1 -- 172.21.15.32:0/4031806217 >> v1:172.21.15.32:6789/0 conn(0x7f4ef0109aa0 legacy=0x7f4ef019d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:42.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:42.105+0000 7f4ef6535700 1 -- 172.21.15.32:0/4031806217 shutdown_connections 2024-04-24T17:27:42.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:42.105+0000 7f4ef6535700 1 -- 172.21.15.32:0/4031806217 >> 172.21.15.32:0/4031806217 conn(0x7f4ef007ceb0 msgr2=0x7f4ef01074a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:27:42.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:42.105+0000 7f4ef6535700 1 -- 172.21.15.32:0/4031806217 shutdown_connections 2024-04-24T17:27:42.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:42.105+0000 7f4ef6535700 1 -- 172.21.15.32:0/4031806217 wait complete. 2024-04-24T17:27:42.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:42.105+0000 7f4ef6535700 1 librados: shutdown 2024-04-24T17:27:42.829 INFO:teuthology.orchestra.run.smithi032.stdout:Generating ssh key... 2024-04-24T17:27:43.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:42 smithi032 bash[21457]: audit 2024-04-24T17:27:42.106625+0000 mgr.y (mgr.14120) 9 : audit [DBG] from='client.14134 v1:172.21.15.32:0/4031806217' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T17:27:43.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:42 smithi032 bash[21457]: audit 2024-04-24T17:27:42.106625+0000 mgr.y (mgr.14120) 9 : audit [DBG] from='client.14134 v1:172.21.15.32:0/4031806217' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T17:27:45.002 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:44 smithi032 bash[21712]: Generating public/private rsa key pair. 2024-04-24T17:27:45.002 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:44 smithi032 bash[21712]: Your identification has been saved in /tmp/tmp0yprve23/key. 2024-04-24T17:27:45.002 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:44 smithi032 bash[21712]: Your public key has been saved in /tmp/tmp0yprve23/key.pub. 2024-04-24T17:27:45.002 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:44 smithi032 bash[21712]: The key fingerprint is: 2024-04-24T17:27:45.002 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:44 smithi032 bash[21712]: SHA256:nnOIrX8S4KnfO7resb9a6X8llnUAsScJyEirZTE0Cp0 ceph-a7981b84-025f-11ef-bc93-c7b262605968 2024-04-24T17:27:45.002 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:44 smithi032 bash[21712]: The key's randomart image is: 2024-04-24T17:27:45.002 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:44 smithi032 bash[21712]: +---[RSA 3072]----+ 2024-04-24T17:27:45.002 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:44 smithi032 bash[21712]: | .. +*o .. oo | 2024-04-24T17:27:45.002 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:44 smithi032 bash[21712]: | .E..=o . o. | 2024-04-24T17:27:45.002 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:44 smithi032 bash[21712]: | . + + .. | 2024-04-24T17:27:45.003 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:44 smithi032 bash[21712]: | +. o o| 2024-04-24T17:27:45.003 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:44 smithi032 bash[21712]: | .. oS o.| 2024-04-24T17:27:45.003 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:44 smithi032 bash[21712]: | o+.o . + .| 2024-04-24T17:27:45.003 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:44 smithi032 bash[21712]: | .. B.+ . o | 2024-04-24T17:27:45.003 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:44 smithi032 bash[21712]: | . +oB. . | 2024-04-24T17:27:45.003 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:44 smithi032 bash[21712]: | o*+BB+o.. | 2024-04-24T17:27:45.003 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:27:44 smithi032 bash[21712]: +----[SHA256]-----+ 2024-04-24T17:27:45.321 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.733+0000 7f32999df700 1 Processor -- start 2024-04-24T17:27:45.321 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.733+0000 7f32999df700 1 -- start start 2024-04-24T17:27:45.321 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.733+0000 7f32999df700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f329410bcc0 con 0x7f3294107890 2024-04-24T17:27:45.321 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.733+0000 7f3292ffd700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3294107890 0x7f3294107cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:46724/0 (socket says 172.21.15.32:46724) 2024-04-24T17:27:45.321 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.733+0000 7f3292ffd700 1 -- 172.21.15.32:0/1888693320 learned_addr learned my addr 172.21.15.32:0/1888693320 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:27:45.321 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.733+0000 7f3291ffb700 1 -- 172.21.15.32:0/1888693320 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4231493371 0 0) 0x7f329410bcc0 con 0x7f3294107890 2024-04-24T17:27:45.322 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.733+0000 7f3291ffb700 1 -- 172.21.15.32:0/1888693320 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f327c0035b0 con 0x7f3294107890 2024-04-24T17:27:45.322 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.733+0000 7f3291ffb700 1 -- 172.21.15.32:0/1888693320 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4234576924 0 0) 0x7f327c0035b0 con 0x7f3294107890 2024-04-24T17:27:45.322 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.733+0000 7f3291ffb700 1 -- 172.21.15.32:0/1888693320 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f329410cea0 con 0x7f3294107890 2024-04-24T17:27:45.322 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.733+0000 7f3291ffb700 1 -- 172.21.15.32:0/1888693320 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f3284001580 con 0x7f3294107890 2024-04-24T17:27:45.322 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.733+0000 7f3291ffb700 1 -- 172.21.15.32:0/1888693320 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(25 keys) ==== 1011+0+0 (unknown 2148541123 0 0) 0x7f3284001c40 con 0x7f3294107890 2024-04-24T17:27:45.322 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.733+0000 7f3291ffb700 1 -- 172.21.15.32:0/1888693320 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f3284005750 con 0x7f3294107890 2024-04-24T17:27:45.322 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.733+0000 7f32999df700 1 -- 172.21.15.32:0/1888693320 >> v1:172.21.15.32:6789/0 conn(0x7f3294107890 legacy=0x7f3294107cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:45.322 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.733+0000 7f32999df700 1 -- 172.21.15.32:0/1888693320 shutdown_connections 2024-04-24T17:27:45.322 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.733+0000 7f32999df700 1 -- 172.21.15.32:0/1888693320 >> 172.21.15.32:0/1888693320 conn(0x7f3294102e30 msgr2=0x7f3294105270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:27:45.322 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.733+0000 7f32999df700 1 -- 172.21.15.32:0/1888693320 shutdown_connections 2024-04-24T17:27:45.322 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.733+0000 7f32999df700 1 -- 172.21.15.32:0/1888693320 wait complete. 2024-04-24T17:27:45.323 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.733+0000 7f32999df700 1 librados: starting msgr at 2024-04-24T17:27:45.323 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.733+0000 7f32999df700 1 librados: starting objecter 2024-04-24T17:27:45.323 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.733+0000 7f32999df700 1 Processor -- start 2024-04-24T17:27:45.323 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.733+0000 7f32999df700 1 -- start start 2024-04-24T17:27:45.323 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.733+0000 7f32999df700 1 librados: setting wanted keys 2024-04-24T17:27:45.323 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.733+0000 7f32999df700 1 librados: calling monclient init 2024-04-24T17:27:45.323 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.733+0000 7f32999df700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f329419bb20 con 0x7f3294107890 2024-04-24T17:27:45.323 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.737+0000 7f3292ffd700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3294107890 0x7f329419b410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:46736/0 (socket says 172.21.15.32:46736) 2024-04-24T17:27:45.323 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.737+0000 7f3292ffd700 1 -- 172.21.15.32:0/1754314584 learned_addr learned my addr 172.21.15.32:0/1754314584 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:27:45.323 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.737+0000 7f32989dd700 10 client.?.objecter ms_handle_connect 0x7f3294107890 2024-04-24T17:27:45.323 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.737+0000 7f32989dd700 10 client.?.objecter resend_mon_ops 2024-04-24T17:27:45.323 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.737+0000 7f32989dd700 1 -- 172.21.15.32:0/1754314584 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3467070913 0 0) 0x7f329419bb20 con 0x7f3294107890 2024-04-24T17:27:45.323 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.737+0000 7f32989dd700 1 -- 172.21.15.32:0/1754314584 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f327c0035b0 con 0x7f3294107890 2024-04-24T17:27:45.323 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.737+0000 7f32989dd700 1 -- 172.21.15.32:0/1754314584 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3464610151 0 0) 0x7f327c0035b0 con 0x7f3294107890 2024-04-24T17:27:45.324 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.737+0000 7f32989dd700 1 -- 172.21.15.32:0/1754314584 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f329419bb20 con 0x7f3294107890 2024-04-24T17:27:45.324 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.737+0000 7f32989dd700 1 -- 172.21.15.32:0/1754314584 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f3284001080 con 0x7f3294107890 2024-04-24T17:27:45.324 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.737+0000 7f32989dd700 1 -- 172.21.15.32:0/1754314584 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2465419950 0 0) 0x7f329419bb20 con 0x7f3294107890 2024-04-24T17:27:45.324 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.737+0000 7f32989dd700 1 -- 172.21.15.32:0/1754314584 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f329419bcf0 con 0x7f3294107890 2024-04-24T17:27:45.324 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.737+0000 7f32999df700 1 -- 172.21.15.32:0/1754314584 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f329419bfe0 con 0x7f3294107890 2024-04-24T17:27:45.324 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.737+0000 7f32999df700 10 client.14136.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-24T17:27:45.324 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.737+0000 7f32999df700 1 -- 172.21.15.32:0/1754314584 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f329419fb30 con 0x7f3294107890 2024-04-24T17:27:45.324 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.737+0000 7f32999df700 1 librados: init done 2024-04-24T17:27:45.324 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.737+0000 7f32989dd700 1 -- 172.21.15.32:0/1754314584 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(25 keys) ==== 1011+0+0 (unknown 2148541123 0 0) 0x7f32840056c0 con 0x7f3294107890 2024-04-24T17:27:45.324 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.737+0000 7f32989dd700 1 -- 172.21.15.32:0/1754314584 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f3284005ef0 con 0x7f3294107890 2024-04-24T17:27:45.324 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.737+0000 7f32989dd700 1 -- 172.21.15.32:0/1754314584 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 8) ==== 46523+0+0 (unknown 1814385894 0 0) 0x7f3284011660 con 0x7f3294107890 2024-04-24T17:27:45.324 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.737+0000 7f32999df700 1 -- 172.21.15.32:0/1754314584 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3280005150 con 0x7f3294107890 2024-04-24T17:27:45.324 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.737+0000 7f32989dd700 1 -- 172.21.15.32:0/1754314584 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 3338368936 0 0) 0x7f3284047fd0 con 0x7f3294107890 2024-04-24T17:27:45.324 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.737+0000 7f32989dd700 10 client.14136.objecter ms_dispatch 0x7f3294001550 osd_map(2..2 src has 1..2) 2024-04-24T17:27:45.324 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.737+0000 7f32989dd700 3 client.14136.objecter handle_osd_map got epochs [2,2] > 0 2024-04-24T17:27:45.325 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.737+0000 7f32989dd700 3 client.14136.objecter handle_osd_map decoding full epoch 2 2024-04-24T17:27:45.325 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.737+0000 7f32989dd700 20 client.14136.objecter dump_active .. 0 homeless 2024-04-24T17:27:45.325 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.737+0000 7f32989dd700 10 client.14136.objecter ms_handle_connect 0x7f327c03c440 2024-04-24T17:27:45.325 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:43.741+0000 7f32989dd700 1 -- 172.21.15.32:0/1754314584 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188728 (unknown 1092875540 0 2656817166) 0x7f328401f290 con 0x7f3294107890 2024-04-24T17:27:45.325 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:44.125+0000 7f32999df700 1 -- 172.21.15.32:0/1754314584 --> v1:172.21.15.32:6800/3410850029 -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f3280000bc0 con 0x7f327c03c440 2024-04-24T17:27:45.325 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:44.657+0000 7f32989dd700 1 -- 172.21.15.32:0/1754314584 <== mgr.14120 v1:172.21.15.32:6800/3410850029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (unknown 0 0 0) 0x7f3280000bc0 con 0x7f327c03c440 2024-04-24T17:27:45.325 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:44.661+0000 7f32999df700 10 librados: watch_flush enter 2024-04-24T17:27:45.325 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:44.661+0000 7f32999df700 10 librados: watch_flush exit 2024-04-24T17:27:45.325 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:44.661+0000 7f32999df700 20 client.14136.objecter shutdown clearing up homeless session... 2024-04-24T17:27:45.325 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:44.661+0000 7f32999df700 10 client.14136.objecter successfully canceled tick 2024-04-24T17:27:45.325 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:44.661+0000 7f32999df700 1 -- 172.21.15.32:0/1754314584 >> v1:172.21.15.32:6800/3410850029 conn(0x7f327c03c440 legacy=0x7f327c03e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:45.325 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:44.661+0000 7f32999df700 1 -- 172.21.15.32:0/1754314584 >> v1:172.21.15.32:6789/0 conn(0x7f3294107890 legacy=0x7f329419b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:45.325 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:44.661+0000 7f32999df700 1 -- 172.21.15.32:0/1754314584 shutdown_connections 2024-04-24T17:27:45.325 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:44.661+0000 7f32999df700 1 -- 172.21.15.32:0/1754314584 >> 172.21.15.32:0/1754314584 conn(0x7f3294102e30 msgr2=0x7f3294105270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:27:45.325 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:44.661+0000 7f32999df700 1 -- 172.21.15.32:0/1754314584 shutdown_connections 2024-04-24T17:27:45.326 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:44.661+0000 7f32999df700 1 -- 172.21.15.32:0/1754314584 wait complete. 2024-04-24T17:27:45.326 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:44.661+0000 7f32999df700 1 librados: shutdown 2024-04-24T17:27:45.989 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:45 smithi032 bash[21457]: audit 2024-04-24T17:27:44.128419+0000 mgr.y (mgr.14120) 10 : audit [DBG] from='client.14136 v1:172.21.15.32:0/1754314584' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T17:27:45.989 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:45 smithi032 bash[21457]: audit 2024-04-24T17:27:44.128419+0000 mgr.y (mgr.14120) 10 : audit [DBG] from='client.14136 v1:172.21.15.32:0/1754314584' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T17:27:45.989 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:45 smithi032 bash[21457]: cephadm 2024-04-24T17:27:44.128633+0000 mgr.y (mgr.14120) 11 : cephadm [INF] Generating ssh key... 2024-04-24T17:27:45.989 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:45 smithi032 bash[21457]: cephadm 2024-04-24T17:27:44.128633+0000 mgr.y (mgr.14120) 11 : cephadm [INF] Generating ssh key... 2024-04-24T17:27:45.989 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:45 smithi032 bash[21457]: audit 2024-04-24T17:27:44.654777+0000 mon.a (mon.0) 63 : audit [INF] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' 2024-04-24T17:27:45.989 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:45 smithi032 bash[21457]: audit 2024-04-24T17:27:44.654777+0000 mon.a (mon.0) 63 : audit [INF] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' 2024-04-24T17:27:45.990 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:45 smithi032 bash[21457]: audit 2024-04-24T17:27:44.659595+0000 mon.a (mon.0) 64 : audit [INF] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' 2024-04-24T17:27:45.990 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:45 smithi032 bash[21457]: audit 2024-04-24T17:27:44.659595+0000 mon.a (mon.0) 64 : audit [INF] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' 2024-04-24T17:27:47.243 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDWf4PvC0e2ZX4vQ31byvDugW7gt2B+a2H2ImNvG/O3jM0WV7pTG/P+8oINdEVDHs1JOFjdNZ+ao8GjgObqNonUX0RtfYTqVoekHG2JA2AYl8Qaj8rm8xmVuyrjizHkxvWiDw1URlh2EvUTRD7SpedPsRAcS2bYDzn99wlptBAN8G/LfNnvqa0eabAb7IA7eqHmyp5mRAI0YfB/jOTpEsI5RWpdSJY43Xvj7oIjXcuBzyCey69M57Zm4XP1LW7EE+DZ0OZsNJGEZGxRvKgshMEBtVspgLV4DkTvTxrD4yocYGknzk3z2DAuiUSLeDuyjJT/J/Zp2LsFNR7d3nOpN0AfF7FU2lJG/CefWEJCZqBDfr6Kbr5AcMvUN/spfV8LobKS/QLlo3sDyBPvanrfCRJHpCSoFlCcQdMF9Tj90IMUuQa3vUceZZ3CPSS4xfF/WEMRsQQQALuL6vgSW5vkqx53Xo5fIrOLrn3q+Arh/TLUlI2FrJEpRt3xXPLOQE7JRrk= ceph-a7981b84-025f-11ef-bc93-c7b262605968 2024-04-24T17:27:47.243 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.153+0000 7fa9e4f34700 1 Processor -- start 2024-04-24T17:27:47.243 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.153+0000 7fa9e4f34700 1 -- start start 2024-04-24T17:27:47.243 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.157+0000 7fa9e4f34700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa9e010bcd0 con 0x7fa9e01078a0 2024-04-24T17:27:47.244 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.157+0000 7fa9de59c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa9e01078a0 0x7fa9e0107cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37424/0 (socket says 172.21.15.32:37424) 2024-04-24T17:27:47.244 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.157+0000 7fa9de59c700 1 -- 172.21.15.32:0/646994971 learned_addr learned my addr 172.21.15.32:0/646994971 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:27:47.244 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.157+0000 7fa9dd59a700 1 -- 172.21.15.32:0/646994971 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3939082901 0 0) 0x7fa9e010bcd0 con 0x7fa9e01078a0 2024-04-24T17:27:47.244 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.157+0000 7fa9dd59a700 1 -- 172.21.15.32:0/646994971 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9c80035b0 con 0x7fa9e01078a0 2024-04-24T17:27:47.244 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.157+0000 7fa9dd59a700 1 -- 172.21.15.32:0/646994971 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 884547008 0 0) 0x7fa9c80035b0 con 0x7fa9e01078a0 2024-04-24T17:27:47.244 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.157+0000 7fa9dd59a700 1 -- 172.21.15.32:0/646994971 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9e010ceb0 con 0x7fa9e01078a0 2024-04-24T17:27:47.244 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.157+0000 7fa9dd59a700 1 -- 172.21.15.32:0/646994971 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fa9d0001580 con 0x7fa9e01078a0 2024-04-24T17:27:47.244 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.157+0000 7fa9dd59a700 1 -- 172.21.15.32:0/646994971 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(25 keys) ==== 1011+0+0 (unknown 2148541123 0 0) 0x7fa9d0001c40 con 0x7fa9e01078a0 2024-04-24T17:27:47.244 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.157+0000 7fa9dd59a700 1 -- 172.21.15.32:0/646994971 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fa9d0005750 con 0x7fa9e01078a0 2024-04-24T17:27:47.244 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.157+0000 7fa9e4f34700 1 -- 172.21.15.32:0/646994971 >> v1:172.21.15.32:6789/0 conn(0x7fa9e01078a0 legacy=0x7fa9e0107cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:47.244 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.157+0000 7fa9e4f34700 1 -- 172.21.15.32:0/646994971 shutdown_connections 2024-04-24T17:27:47.244 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.157+0000 7fa9e4f34700 1 -- 172.21.15.32:0/646994971 >> 172.21.15.32:0/646994971 conn(0x7fa9e0102e40 msgr2=0x7fa9e0105280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:27:47.244 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.157+0000 7fa9e4f34700 1 -- 172.21.15.32:0/646994971 shutdown_connections 2024-04-24T17:27:47.244 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.157+0000 7fa9e4f34700 1 -- 172.21.15.32:0/646994971 wait complete. 2024-04-24T17:27:47.245 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.157+0000 7fa9e4f34700 1 librados: starting msgr at 2024-04-24T17:27:47.245 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.157+0000 7fa9e4f34700 1 librados: starting objecter 2024-04-24T17:27:47.245 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.157+0000 7fa9e4f34700 1 Processor -- start 2024-04-24T17:27:47.245 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.157+0000 7fa9e4f34700 1 -- start start 2024-04-24T17:27:47.245 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.157+0000 7fa9e4f34700 1 librados: setting wanted keys 2024-04-24T17:27:47.245 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.157+0000 7fa9e4f34700 1 librados: calling monclient init 2024-04-24T17:27:47.245 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.157+0000 7fa9e4f34700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa9e019bb50 con 0x7fa9e01078a0 2024-04-24T17:27:47.245 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.157+0000 7fa9de59c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa9e01078a0 0x7fa9e019b440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37438/0 (socket says 172.21.15.32:37438) 2024-04-24T17:27:47.245 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.157+0000 7fa9de59c700 1 -- 172.21.15.32:0/2120402265 learned_addr learned my addr 172.21.15.32:0/2120402265 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:27:47.245 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.157+0000 7fa9cf7fe700 10 client.?.objecter ms_handle_connect 0x7fa9e01078a0 2024-04-24T17:27:47.245 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.157+0000 7fa9cf7fe700 10 client.?.objecter resend_mon_ops 2024-04-24T17:27:47.245 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.157+0000 7fa9cf7fe700 1 -- 172.21.15.32:0/2120402265 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2407774021 0 0) 0x7fa9e019bb50 con 0x7fa9e01078a0 2024-04-24T17:27:47.245 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.157+0000 7fa9cf7fe700 1 -- 172.21.15.32:0/2120402265 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9c80035b0 con 0x7fa9e01078a0 2024-04-24T17:27:47.245 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.157+0000 7fa9cf7fe700 1 -- 172.21.15.32:0/2120402265 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 427803235 0 0) 0x7fa9c80035b0 con 0x7fa9e01078a0 2024-04-24T17:27:47.245 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.157+0000 7fa9cf7fe700 1 -- 172.21.15.32:0/2120402265 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa9e019bb50 con 0x7fa9e01078a0 2024-04-24T17:27:47.246 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.157+0000 7fa9cf7fe700 1 -- 172.21.15.32:0/2120402265 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fa9d0001080 con 0x7fa9e01078a0 2024-04-24T17:27:47.246 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.157+0000 7fa9cf7fe700 1 -- 172.21.15.32:0/2120402265 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1925997866 0 0) 0x7fa9e019bb50 con 0x7fa9e01078a0 2024-04-24T17:27:47.246 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.157+0000 7fa9cf7fe700 1 -- 172.21.15.32:0/2120402265 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9e019bd20 con 0x7fa9e01078a0 2024-04-24T17:27:47.246 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.157+0000 7fa9e4f34700 1 -- 172.21.15.32:0/2120402265 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa9e019c010 con 0x7fa9e01078a0 2024-04-24T17:27:47.246 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.157+0000 7fa9e4f34700 10 client.14138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-24T17:27:47.246 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.161+0000 7fa9e4f34700 1 -- 172.21.15.32:0/2120402265 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa9e019fb60 con 0x7fa9e01078a0 2024-04-24T17:27:47.246 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.161+0000 7fa9e4f34700 1 librados: init done 2024-04-24T17:27:47.246 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.161+0000 7fa9cf7fe700 1 -- 172.21.15.32:0/2120402265 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(25 keys) ==== 1011+0+0 (unknown 2148541123 0 0) 0x7fa9d0005680 con 0x7fa9e01078a0 2024-04-24T17:27:47.246 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.161+0000 7fa9cf7fe700 1 -- 172.21.15.32:0/2120402265 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fa9d0005ed0 con 0x7fa9e01078a0 2024-04-24T17:27:47.246 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.161+0000 7fa9cf7fe700 1 -- 172.21.15.32:0/2120402265 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 8) ==== 46523+0+0 (unknown 1814385894 0 0) 0x7fa9d0011640 con 0x7fa9e01078a0 2024-04-24T17:27:47.246 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.161+0000 7fa9e4f34700 1 -- 172.21.15.32:0/2120402265 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9c0005150 con 0x7fa9e01078a0 2024-04-24T17:27:47.246 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.161+0000 7fa9cf7fe700 1 -- 172.21.15.32:0/2120402265 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 3338368936 0 0) 0x7fa9d0047ee0 con 0x7fa9e01078a0 2024-04-24T17:27:47.246 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.161+0000 7fa9cf7fe700 10 client.14138.objecter ms_dispatch 0x7fa9e0001550 osd_map(2..2 src has 1..2) 2024-04-24T17:27:47.247 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.161+0000 7fa9cf7fe700 3 client.14138.objecter handle_osd_map got epochs [2,2] > 0 2024-04-24T17:27:47.247 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.161+0000 7fa9cf7fe700 3 client.14138.objecter handle_osd_map decoding full epoch 2 2024-04-24T17:27:47.247 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.161+0000 7fa9cf7fe700 20 client.14138.objecter dump_active .. 0 homeless 2024-04-24T17:27:47.247 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.161+0000 7fa9cf7fe700 10 client.14138.objecter ms_handle_connect 0x7fa9c803c440 2024-04-24T17:27:47.247 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.165+0000 7fa9cf7fe700 1 -- 172.21.15.32:0/2120402265 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188728 (unknown 1092875540 0 2656817166) 0x7fa9d0017550 con 0x7fa9e01078a0 2024-04-24T17:27:47.247 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.553+0000 7fa9e4f34700 1 -- 172.21.15.32:0/2120402265 --> v1:172.21.15.32:6800/3410850029 -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7fa9c0000bc0 con 0x7fa9c803c440 2024-04-24T17:27:47.247 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.553+0000 7fa9cf7fe700 1 -- 172.21.15.32:0/2120402265 <== mgr.14120 v1:172.21.15.32:6800/3410850029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (unknown 0 0 4159336062) 0x7fa9c0000bc0 con 0x7fa9c803c440 2024-04-24T17:27:47.247 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.553+0000 7fa9e4f34700 10 librados: watch_flush enter 2024-04-24T17:27:47.247 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.553+0000 7fa9e4f34700 10 librados: watch_flush exit 2024-04-24T17:27:47.247 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.553+0000 7fa9e4f34700 20 client.14138.objecter shutdown clearing up homeless session... 2024-04-24T17:27:47.247 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.553+0000 7fa9e4f34700 10 client.14138.objecter successfully canceled tick 2024-04-24T17:27:47.247 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.553+0000 7fa9e4f34700 1 -- 172.21.15.32:0/2120402265 >> v1:172.21.15.32:6800/3410850029 conn(0x7fa9c803c440 legacy=0x7fa9c803e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:47.248 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.553+0000 7fa9e4f34700 1 -- 172.21.15.32:0/2120402265 >> v1:172.21.15.32:6789/0 conn(0x7fa9e01078a0 legacy=0x7fa9e019b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:47.248 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.553+0000 7fa9e4f34700 1 -- 172.21.15.32:0/2120402265 shutdown_connections 2024-04-24T17:27:47.248 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.553+0000 7fa9e4f34700 1 -- 172.21.15.32:0/2120402265 >> 172.21.15.32:0/2120402265 conn(0x7fa9e0102e40 msgr2=0x7fa9e0105280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:27:47.248 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.553+0000 7fa9e4f34700 1 -- 172.21.15.32:0/2120402265 shutdown_connections 2024-04-24T17:27:47.248 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.553+0000 7fa9e4f34700 1 -- 172.21.15.32:0/2120402265 wait complete. 2024-04-24T17:27:47.248 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:46.553+0000 7fa9e4f34700 1 librados: shutdown 2024-04-24T17:27:47.248 INFO:teuthology.orchestra.run.smithi032.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-24T17:27:47.248 INFO:teuthology.orchestra.run.smithi032.stdout:Adding key to root@localhost authorized_keys... 2024-04-24T17:27:47.248 INFO:teuthology.orchestra.run.smithi032.stdout:Adding host smithi032... 2024-04-24T17:27:47.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:46 smithi032 bash[21457]: audit 2024-04-24T17:27:46.555437+0000 mgr.y (mgr.14120) 12 : audit [DBG] from='client.14138 v1:172.21.15.32:0/2120402265' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T17:27:47.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:46 smithi032 bash[21457]: audit 2024-04-24T17:27:46.555437+0000 mgr.y (mgr.14120) 12 : audit [DBG] from='client.14138 v1:172.21.15.32:0/2120402265' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T17:27:48.962 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:48 smithi032 bash[21457]: audit 2024-04-24T17:27:48.610630+0000 mgr.y (mgr.14120) 13 : audit [DBG] from='client.14140 v1:172.21.15.32:0/2637178815' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi032", "addr": "172.21.15.32", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T17:27:48.962 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:48 smithi032 bash[21457]: audit 2024-04-24T17:27:48.610630+0000 mgr.y (mgr.14120) 13 : audit [DBG] from='client.14140 v1:172.21.15.32:0/2637178815' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi032", "addr": "172.21.15.32", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T17:27:50.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:49 smithi032 bash[21457]: cephadm 2024-04-24T17:27:49.307910+0000 mgr.y (mgr.14120) 14 : cephadm [INF] Deploying cephadm binary to smithi032 2024-04-24T17:27:50.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:49 smithi032 bash[21457]: cephadm 2024-04-24T17:27:49.307910+0000 mgr.y (mgr.14120) 14 : cephadm [INF] Deploying cephadm binary to smithi032 2024-04-24T17:27:55.167 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout Added host 'smithi032' with addr '172.21.15.32' 2024-04-24T17:27:55.167 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.200+0000 7fa9b271d700 1 Processor -- start 2024-04-24T17:27:55.168 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.200+0000 7fa9b271d700 1 -- start start 2024-04-24T17:27:55.168 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.200+0000 7fa9b271d700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa9ac07ae80 con 0x7fa9ac1082f0 2024-04-24T17:27:55.168 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.200+0000 7fa9abfff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa9ac1082f0 0x7fa9ac10a6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37440/0 (socket says 172.21.15.32:37440) 2024-04-24T17:27:55.168 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.200+0000 7fa9abfff700 1 -- 172.21.15.32:0/200117677 learned_addr learned my addr 172.21.15.32:0/200117677 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:27:55.168 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.200+0000 7fa9ab7fe700 1 -- 172.21.15.32:0/200117677 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2484851509 0 0) 0x7fa9ac07ae80 con 0x7fa9ac1082f0 2024-04-24T17:27:55.168 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.200+0000 7fa9ab7fe700 1 -- 172.21.15.32:0/200117677 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9980035b0 con 0x7fa9ac1082f0 2024-04-24T17:27:55.168 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.200+0000 7fa9ab7fe700 1 -- 172.21.15.32:0/200117677 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1176860603 0 0) 0x7fa9980035b0 con 0x7fa9ac1082f0 2024-04-24T17:27:55.168 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.200+0000 7fa9ab7fe700 1 -- 172.21.15.32:0/200117677 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9ac10ce10 con 0x7fa9ac1082f0 2024-04-24T17:27:55.168 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.200+0000 7fa9ab7fe700 1 -- 172.21.15.32:0/200117677 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fa99c001900 con 0x7fa9ac1082f0 2024-04-24T17:27:55.168 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.200+0000 7fa9ab7fe700 1 -- 172.21.15.32:0/200117677 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(25 keys) ==== 1011+0+0 (unknown 2148541123 0 0) 0x7fa99c002090 con 0x7fa9ac1082f0 2024-04-24T17:27:55.168 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.200+0000 7fa9ab7fe700 1 -- 172.21.15.32:0/200117677 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fa99c005ce0 con 0x7fa9ac1082f0 2024-04-24T17:27:55.168 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.204+0000 7fa9b271d700 1 -- 172.21.15.32:0/200117677 >> v1:172.21.15.32:6789/0 conn(0x7fa9ac1082f0 legacy=0x7fa9ac10a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:55.168 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.204+0000 7fa9b271d700 1 -- 172.21.15.32:0/200117677 shutdown_connections 2024-04-24T17:27:55.168 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.204+0000 7fa9b271d700 1 -- 172.21.15.32:0/200117677 >> 172.21.15.32:0/200117677 conn(0x7fa9ac101d00 msgr2=0x7fa9ac104160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:27:55.169 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.204+0000 7fa9b271d700 1 -- 172.21.15.32:0/200117677 shutdown_connections 2024-04-24T17:27:55.169 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.204+0000 7fa9b271d700 1 -- 172.21.15.32:0/200117677 wait complete. 2024-04-24T17:27:55.169 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.204+0000 7fa9b271d700 1 librados: starting msgr at 2024-04-24T17:27:55.169 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.204+0000 7fa9b271d700 1 librados: starting objecter 2024-04-24T17:27:55.169 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.204+0000 7fa9b271d700 1 Processor -- start 2024-04-24T17:27:55.169 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.204+0000 7fa9b271d700 1 -- start start 2024-04-24T17:27:55.169 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.204+0000 7fa9b271d700 1 librados: setting wanted keys 2024-04-24T17:27:55.169 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.204+0000 7fa9b271d700 1 librados: calling monclient init 2024-04-24T17:27:55.169 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.204+0000 7fa9b271d700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa9ac1a1570 con 0x7fa9ac1082f0 2024-04-24T17:27:55.169 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.204+0000 7fa9abfff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa9ac1082f0 0x7fa9ac107b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37446/0 (socket says 172.21.15.32:37446) 2024-04-24T17:27:55.169 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.204+0000 7fa9abfff700 1 -- 172.21.15.32:0/2637178815 learned_addr learned my addr 172.21.15.32:0/2637178815 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:27:55.169 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.204+0000 7fa9a9ffb700 10 client.?.objecter ms_handle_connect 0x7fa9ac1082f0 2024-04-24T17:27:55.169 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.204+0000 7fa9a9ffb700 10 client.?.objecter resend_mon_ops 2024-04-24T17:27:55.169 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.204+0000 7fa9a9ffb700 1 -- 172.21.15.32:0/2637178815 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3834949880 0 0) 0x7fa9ac1a1570 con 0x7fa9ac1082f0 2024-04-24T17:27:55.170 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.204+0000 7fa9a9ffb700 1 -- 172.21.15.32:0/2637178815 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9980056e0 con 0x7fa9ac1082f0 2024-04-24T17:27:55.170 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.204+0000 7fa9a9ffb700 1 -- 172.21.15.32:0/2637178815 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1929965565 0 0) 0x7fa9ac10ce10 con 0x7fa9ac1082f0 2024-04-24T17:27:55.170 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.204+0000 7fa9a9ffb700 1 -- 172.21.15.32:0/2637178815 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa9ac1a1570 con 0x7fa9ac1082f0 2024-04-24T17:27:55.170 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.204+0000 7fa9a9ffb700 1 -- 172.21.15.32:0/2637178815 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fa99c004bf0 con 0x7fa9ac1082f0 2024-04-24T17:27:55.170 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.204+0000 7fa9a9ffb700 1 -- 172.21.15.32:0/2637178815 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4009183671 0 0) 0x7fa9ac1a1570 con 0x7fa9ac1082f0 2024-04-24T17:27:55.170 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.204+0000 7fa9a9ffb700 1 -- 172.21.15.32:0/2637178815 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9ac1a2750 con 0x7fa9ac1082f0 2024-04-24T17:27:55.170 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.204+0000 7fa9b271d700 1 -- 172.21.15.32:0/2637178815 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa9ac1a17a0 con 0x7fa9ac1082f0 2024-04-24T17:27:55.170 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.204+0000 7fa9b271d700 10 client.14140.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-24T17:27:55.170 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.204+0000 7fa9b271d700 1 -- 172.21.15.32:0/2637178815 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa9ac1a1ce0 con 0x7fa9ac1082f0 2024-04-24T17:27:55.170 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.204+0000 7fa9b271d700 1 librados: init done 2024-04-24T17:27:55.170 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.204+0000 7fa9a9ffb700 1 -- 172.21.15.32:0/2637178815 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(25 keys) ==== 1011+0+0 (unknown 2148541123 0 0) 0x7fa99c001f00 con 0x7fa9ac1082f0 2024-04-24T17:27:55.170 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.204+0000 7fa9a9ffb700 1 -- 172.21.15.32:0/2637178815 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fa99c006230 con 0x7fa9ac1082f0 2024-04-24T17:27:55.170 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.204+0000 7fa9a9ffb700 1 -- 172.21.15.32:0/2637178815 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 8) ==== 46523+0+0 (unknown 1814385894 0 0) 0x7fa99c0119c0 con 0x7fa9ac1082f0 2024-04-24T17:27:55.171 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.204+0000 7fa9a9ffb700 1 -- 172.21.15.32:0/2637178815 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 3338368936 0 0) 0x7fa99c048230 con 0x7fa9ac1082f0 2024-04-24T17:27:55.171 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.204+0000 7fa9a9ffb700 10 client.14140.objecter ms_dispatch 0x7fa9ac001550 osd_map(2..2 src has 1..2) 2024-04-24T17:27:55.171 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.204+0000 7fa9a9ffb700 3 client.14140.objecter handle_osd_map got epochs [2,2] > 0 2024-04-24T17:27:55.171 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.204+0000 7fa9a9ffb700 3 client.14140.objecter handle_osd_map decoding full epoch 2 2024-04-24T17:27:55.171 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.204+0000 7fa9b271d700 1 -- 172.21.15.32:0/2637178815 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa98c005120 con 0x7fa9ac1082f0 2024-04-24T17:27:55.171 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.204+0000 7fa9a9ffb700 20 client.14140.objecter dump_active .. 0 homeless 2024-04-24T17:27:55.171 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.204+0000 7fa9a9ffb700 10 client.14140.objecter ms_handle_connect 0x7fa99803c220 2024-04-24T17:27:55.171 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.212+0000 7fa9a9ffb700 1 -- 172.21.15.32:0/2637178815 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188728 (unknown 1092875540 0 2656817166) 0x7fa9980056e0 con 0x7fa9ac1082f0 2024-04-24T17:27:55.171 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:48.604+0000 7fa9b271d700 1 -- 172.21.15.32:0/2637178815 --> v1:172.21.15.32:6800/3410850029 -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi032", "addr": "172.21.15.32", "target": ["mon-mgr", ""]}) -- 0x7fa98c000b90 con 0x7fa99803c220 2024-04-24T17:27:55.171 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:53.208+0000 7fa9b1f1c700 10 client.14140.objecter tick 2024-04-24T17:27:55.171 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:54.320+0000 7fa9a9ffb700 1 -- 172.21.15.32:0/2637178815 <== mgr.14120 v1:172.21.15.32:6800/3410850029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (unknown 0 0 3278889494) 0x7fa98c000b90 con 0x7fa99803c220 2024-04-24T17:27:55.171 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:54.320+0000 7fa9b271d700 10 librados: watch_flush enter 2024-04-24T17:27:55.171 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:54.320+0000 7fa9b271d700 10 librados: watch_flush exit 2024-04-24T17:27:55.171 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:54.320+0000 7fa9b271d700 20 client.14140.objecter shutdown clearing up homeless session... 2024-04-24T17:27:55.171 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:54.320+0000 7fa9b271d700 10 client.14140.objecter successfully canceled tick 2024-04-24T17:27:55.172 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:54.320+0000 7fa9b271d700 1 -- 172.21.15.32:0/2637178815 >> v1:172.21.15.32:6800/3410850029 conn(0x7fa99803c220 legacy=0x7fa99803e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:55.172 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:54.320+0000 7fa9b271d700 1 -- 172.21.15.32:0/2637178815 >> v1:172.21.15.32:6789/0 conn(0x7fa9ac1082f0 legacy=0x7fa9ac107b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:55.172 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:54.320+0000 7fa9b271d700 1 -- 172.21.15.32:0/2637178815 shutdown_connections 2024-04-24T17:27:55.172 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:54.320+0000 7fa9b271d700 1 -- 172.21.15.32:0/2637178815 >> 172.21.15.32:0/2637178815 conn(0x7fa9ac101d00 msgr2=0x7fa9ac192150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:27:55.172 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:54.320+0000 7fa9b271d700 1 -- 172.21.15.32:0/2637178815 shutdown_connections 2024-04-24T17:27:55.172 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:54.320+0000 7fa9b271d700 1 -- 172.21.15.32:0/2637178815 wait complete. 2024-04-24T17:27:55.172 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:54.320+0000 7fa9b271d700 1 librados: shutdown 2024-04-24T17:27:55.172 INFO:teuthology.orchestra.run.smithi032.stdout:Deploying unmanaged mon service... 2024-04-24T17:27:55.501 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:55 smithi032 bash[21457]: audit 2024-04-24T17:27:54.322108+0000 mon.a (mon.0) 65 : audit [INF] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' 2024-04-24T17:27:55.501 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:55 smithi032 bash[21457]: audit 2024-04-24T17:27:54.322108+0000 mon.a (mon.0) 65 : audit [INF] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' 2024-04-24T17:27:55.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:55 smithi032 bash[21457]: cephadm 2024-04-24T17:27:54.322654+0000 mgr.y (mgr.14120) 15 : cephadm [INF] Added host smithi032 2024-04-24T17:27:55.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:55 smithi032 bash[21457]: cephadm 2024-04-24T17:27:54.322654+0000 mgr.y (mgr.14120) 15 : cephadm [INF] Added host smithi032 2024-04-24T17:27:55.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:55 smithi032 bash[21457]: audit 2024-04-24T17:27:54.345168+0000 mon.a (mon.0) 66 : audit [DBG] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T17:27:55.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:55 smithi032 bash[21457]: audit 2024-04-24T17:27:54.345168+0000 mon.a (mon.0) 66 : audit [DBG] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T17:27:57.904 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-24T17:27:57.905 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.692+0000 7fa0a4f7e700 1 Processor -- start 2024-04-24T17:27:57.905 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.692+0000 7fa0a4f7e700 1 -- start start 2024-04-24T17:27:57.905 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.692+0000 7fa0a4f7e700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa0a007ba90 con 0x7fa0a007ce90 2024-04-24T17:27:57.905 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.692+0000 7fa09e59c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa0a007ce90 0x7fa0a007b2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51506/0 (socket says 172.21.15.32:51506) 2024-04-24T17:27:57.905 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.692+0000 7fa09e59c700 1 -- 172.21.15.32:0/3800402781 learned_addr learned my addr 172.21.15.32:0/3800402781 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:27:57.905 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.692+0000 7fa09d59a700 1 -- 172.21.15.32:0/3800402781 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 551681955 0 0) 0x7fa0a007ba90 con 0x7fa0a007ce90 2024-04-24T17:27:57.905 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.692+0000 7fa09d59a700 1 -- 172.21.15.32:0/3800402781 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa0880035b0 con 0x7fa0a007ce90 2024-04-24T17:27:57.905 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.692+0000 7fa09d59a700 1 -- 172.21.15.32:0/3800402781 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4061546747 0 0) 0x7fa0880035b0 con 0x7fa0a007ce90 2024-04-24T17:27:57.905 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.692+0000 7fa09d59a700 1 -- 172.21.15.32:0/3800402781 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0a007bc60 con 0x7fa0a007ce90 2024-04-24T17:27:57.905 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.692+0000 7fa09d59a700 1 -- 172.21.15.32:0/3800402781 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fa090001580 con 0x7fa0a007ce90 2024-04-24T17:27:57.906 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.692+0000 7fa09d59a700 1 -- 172.21.15.32:0/3800402781 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(25 keys) ==== 1011+0+0 (unknown 2148541123 0 0) 0x7fa090001c40 con 0x7fa0a007ce90 2024-04-24T17:27:57.906 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.692+0000 7fa09d59a700 1 -- 172.21.15.32:0/3800402781 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fa090005960 con 0x7fa0a007ce90 2024-04-24T17:27:57.906 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.692+0000 7fa0a4f7e700 1 -- 172.21.15.32:0/3800402781 >> v1:172.21.15.32:6789/0 conn(0x7fa0a007ce90 legacy=0x7fa0a007b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:57.906 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.692+0000 7fa0a4f7e700 1 -- 172.21.15.32:0/3800402781 shutdown_connections 2024-04-24T17:27:57.906 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.692+0000 7fa0a4f7e700 1 -- 172.21.15.32:0/3800402781 >> 172.21.15.32:0/3800402781 conn(0x7fa0a0102df0 msgr2=0x7fa0a0105230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:27:57.906 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.692+0000 7fa0a4f7e700 1 -- 172.21.15.32:0/3800402781 shutdown_connections 2024-04-24T17:27:57.906 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.692+0000 7fa0a4f7e700 1 -- 172.21.15.32:0/3800402781 wait complete. 2024-04-24T17:27:57.906 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.696+0000 7fa0a4f7e700 1 librados: starting msgr at 2024-04-24T17:27:57.906 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.696+0000 7fa0a4f7e700 1 librados: starting objecter 2024-04-24T17:27:57.906 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.696+0000 7fa0a4f7e700 1 Processor -- start 2024-04-24T17:27:57.906 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.696+0000 7fa0a4f7e700 1 -- start start 2024-04-24T17:27:57.906 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.696+0000 7fa0a4f7e700 1 librados: setting wanted keys 2024-04-24T17:27:57.906 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.696+0000 7fa0a4f7e700 1 librados: calling monclient init 2024-04-24T17:27:57.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.696+0000 7fa0a4f7e700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa0a019dd50 con 0x7fa0a007ce90 2024-04-24T17:27:57.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.696+0000 7fa09e59c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa0a007ce90 0x7fa0a019d640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51516/0 (socket says 172.21.15.32:51516) 2024-04-24T17:27:57.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.696+0000 7fa09e59c700 1 -- 172.21.15.32:0/3835371467 learned_addr learned my addr 172.21.15.32:0/3835371467 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:27:57.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.696+0000 7fa08f7fe700 10 client.?.objecter ms_handle_connect 0x7fa0a007ce90 2024-04-24T17:27:57.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.696+0000 7fa08f7fe700 10 client.?.objecter resend_mon_ops 2024-04-24T17:27:57.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.696+0000 7fa08f7fe700 1 -- 172.21.15.32:0/3835371467 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4152474312 0 0) 0x7fa0a019dd50 con 0x7fa0a007ce90 2024-04-24T17:27:57.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.696+0000 7fa08f7fe700 1 -- 172.21.15.32:0/3835371467 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa0880035b0 con 0x7fa0a007ce90 2024-04-24T17:27:57.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.696+0000 7fa08f7fe700 1 -- 172.21.15.32:0/3835371467 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2699608989 0 0) 0x7fa0880035b0 con 0x7fa0a007ce90 2024-04-24T17:27:57.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.696+0000 7fa08f7fe700 1 -- 172.21.15.32:0/3835371467 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa0a019dd50 con 0x7fa0a007ce90 2024-04-24T17:27:57.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.696+0000 7fa08f7fe700 1 -- 172.21.15.32:0/3835371467 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fa090001080 con 0x7fa0a007ce90 2024-04-24T17:27:57.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.696+0000 7fa08f7fe700 1 -- 172.21.15.32:0/3835371467 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 558919364 0 0) 0x7fa0a019dd50 con 0x7fa0a007ce90 2024-04-24T17:27:57.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.696+0000 7fa08f7fe700 1 -- 172.21.15.32:0/3835371467 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0a019df20 con 0x7fa0a007ce90 2024-04-24T17:27:57.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.696+0000 7fa0a4f7e700 1 -- 172.21.15.32:0/3835371467 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa0a019e210 con 0x7fa0a007ce90 2024-04-24T17:27:57.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.696+0000 7fa0a4f7e700 10 client.14142.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-24T17:27:57.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.696+0000 7fa0a4f7e700 1 -- 172.21.15.32:0/3835371467 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa0a01a1d60 con 0x7fa0a007ce90 2024-04-24T17:27:57.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.696+0000 7fa0a4f7e700 1 librados: init done 2024-04-24T17:27:57.908 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.696+0000 7fa08f7fe700 1 -- 172.21.15.32:0/3835371467 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(25 keys) ==== 1011+0+0 (unknown 2148541123 0 0) 0x7fa090005910 con 0x7fa0a007ce90 2024-04-24T17:27:57.908 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.696+0000 7fa08f7fe700 1 -- 172.21.15.32:0/3835371467 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fa090005fe0 con 0x7fa0a007ce90 2024-04-24T17:27:57.908 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.696+0000 7fa0a4f7e700 1 -- 172.21.15.32:0/3835371467 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa080005150 con 0x7fa0a007ce90 2024-04-24T17:27:57.908 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.696+0000 7fa08f7fe700 1 -- 172.21.15.32:0/3835371467 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 8) ==== 46523+0+0 (unknown 1814385894 0 0) 0x7fa090011750 con 0x7fa0a007ce90 2024-04-24T17:27:57.908 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.696+0000 7fa08f7fe700 1 -- 172.21.15.32:0/3835371467 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 3338368936 0 0) 0x7fa090048240 con 0x7fa0a007ce90 2024-04-24T17:27:57.908 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.696+0000 7fa08f7fe700 10 client.14142.objecter ms_dispatch 0x7fa0a0001550 osd_map(2..2 src has 1..2) 2024-04-24T17:27:57.908 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.696+0000 7fa08f7fe700 3 client.14142.objecter handle_osd_map got epochs [2,2] > 0 2024-04-24T17:27:57.908 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.696+0000 7fa08f7fe700 3 client.14142.objecter handle_osd_map decoding full epoch 2 2024-04-24T17:27:57.908 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.696+0000 7fa08f7fe700 20 client.14142.objecter dump_active .. 0 homeless 2024-04-24T17:27:57.908 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.700+0000 7fa08f7fe700 10 client.14142.objecter ms_handle_connect 0x7fa08803c3c0 2024-04-24T17:27:57.908 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:56.708+0000 7fa08f7fe700 1 -- 172.21.15.32:0/3835371467 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188728 (unknown 1092875540 0 2656817166) 0x7fa090017720 con 0x7fa0a007ce90 2024-04-24T17:27:57.908 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:57.124+0000 7fa0a4f7e700 1 -- 172.21.15.32:0/3835371467 --> v1:172.21.15.32:6800/3410850029 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fa080000bc0 con 0x7fa08803c3c0 2024-04-24T17:27:57.908 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:57.136+0000 7fa08f7fe700 1 -- 172.21.15.32:0/3835371467 <== mgr.14120 v1:172.21.15.32:6800/3410850029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (unknown 0 0 3265049985) 0x7fa080000bc0 con 0x7fa08803c3c0 2024-04-24T17:27:57.908 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:57.136+0000 7fa0a4f7e700 10 librados: watch_flush enter 2024-04-24T17:27:57.909 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:57.136+0000 7fa0a4f7e700 10 librados: watch_flush exit 2024-04-24T17:27:57.909 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:57.136+0000 7fa0a4f7e700 20 client.14142.objecter shutdown clearing up homeless session... 2024-04-24T17:27:57.909 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:57.136+0000 7fa0a4f7e700 10 client.14142.objecter successfully canceled tick 2024-04-24T17:27:57.909 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:57.136+0000 7fa0a4f7e700 1 -- 172.21.15.32:0/3835371467 >> v1:172.21.15.32:6800/3410850029 conn(0x7fa08803c3c0 legacy=0x7fa08803e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:57.909 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:57.136+0000 7fa0a4f7e700 1 -- 172.21.15.32:0/3835371467 >> v1:172.21.15.32:6789/0 conn(0x7fa0a007ce90 legacy=0x7fa0a019d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:57.909 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:57.136+0000 7fa0a4f7e700 1 -- 172.21.15.32:0/3835371467 shutdown_connections 2024-04-24T17:27:57.909 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:57.136+0000 7fa0a4f7e700 1 -- 172.21.15.32:0/3835371467 >> 172.21.15.32:0/3835371467 conn(0x7fa0a0102df0 msgr2=0x7fa0a010bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:27:57.909 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:57.136+0000 7fa0a4f7e700 1 -- 172.21.15.32:0/3835371467 shutdown_connections 2024-04-24T17:27:57.909 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:57.136+0000 7fa0a4f7e700 1 -- 172.21.15.32:0/3835371467 wait complete. 2024-04-24T17:27:57.909 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:57.136+0000 7fa0a4f7e700 1 librados: shutdown 2024-04-24T17:27:57.909 INFO:teuthology.orchestra.run.smithi032.stdout:Deploying unmanaged mgr service... 2024-04-24T17:27:58.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:58 smithi032 bash[21457]: audit 2024-04-24T17:27:57.130119+0000 mgr.y (mgr.14120) 16 : audit [DBG] from='client.14142 v1:172.21.15.32:0/3835371467' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T17:27:58.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:58 smithi032 bash[21457]: audit 2024-04-24T17:27:57.130119+0000 mgr.y (mgr.14120) 16 : audit [DBG] from='client.14142 v1:172.21.15.32:0/3835371467' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T17:27:58.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:58 smithi032 bash[21457]: cephadm 2024-04-24T17:27:57.133508+0000 mgr.y (mgr.14120) 17 : cephadm [INF] Saving service mon spec with placement count:5 2024-04-24T17:27:58.251 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:58 smithi032 bash[21457]: cephadm 2024-04-24T17:27:57.133508+0000 mgr.y (mgr.14120) 17 : cephadm [INF] Saving service mon spec with placement count:5 2024-04-24T17:27:58.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:58 smithi032 bash[21457]: audit 2024-04-24T17:27:57.138502+0000 mon.a (mon.0) 67 : audit [INF] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' 2024-04-24T17:27:58.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:58 smithi032 bash[21457]: audit 2024-04-24T17:27:57.138502+0000 mon.a (mon.0) 67 : audit [INF] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' 2024-04-24T17:27:58.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:58 smithi032 bash[21457]: cluster 2024-04-24T17:27:57.204292+0000 mgr.y (mgr.14120) 18 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:27:58.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:58 smithi032 bash[21457]: cluster 2024-04-24T17:27:57.204292+0000 mgr.y (mgr.14120) 18 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:27:58.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:58 smithi032 bash[21457]: audit 2024-04-24T17:27:57.622462+0000 mon.a (mon.0) 68 : audit [INF] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' 2024-04-24T17:27:58.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:58 smithi032 bash[21457]: audit 2024-04-24T17:27:57.622462+0000 mon.a (mon.0) 68 : audit [INF] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' 2024-04-24T17:27:59.440 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:59 smithi032 bash[21457]: audit 2024-04-24T17:27:58.167586+0000 mon.a (mon.0) 69 : audit [INF] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' 2024-04-24T17:27:59.440 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:27:59 smithi032 bash[21457]: audit 2024-04-24T17:27:58.167586+0000 mon.a (mon.0) 69 : audit [INF] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' 2024-04-24T17:27:59.904 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-24T17:27:59.904 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.796+0000 7f8fff5bf700 1 Processor -- start 2024-04-24T17:27:59.904 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.796+0000 7f8fff5bf700 1 -- start start 2024-04-24T17:27:59.904 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.796+0000 7f8fff5bf700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8ff810ded0 con 0x7f8ff8109aa0 2024-04-24T17:27:59.904 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.796+0000 7f8ffd35b700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f8ff8109aa0 0x7f8ff8109ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51530/0 (socket says 172.21.15.32:51530) 2024-04-24T17:27:59.904 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.796+0000 7f8ffd35b700 1 -- 172.21.15.32:0/3369997358 learned_addr learned my addr 172.21.15.32:0/3369997358 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:27:59.904 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.796+0000 7f8feffff700 1 -- 172.21.15.32:0/3369997358 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 705736832 0 0) 0x7f8ff810ded0 con 0x7f8ff8109aa0 2024-04-24T17:27:59.904 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.796+0000 7f8feffff700 1 -- 172.21.15.32:0/3369997358 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8fe40035b0 con 0x7f8ff8109aa0 2024-04-24T17:27:59.904 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.796+0000 7f8feffff700 1 -- 172.21.15.32:0/3369997358 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1799172428 0 0) 0x7f8fe40035b0 con 0x7f8ff8109aa0 2024-04-24T17:27:59.904 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.796+0000 7f8feffff700 1 -- 172.21.15.32:0/3369997358 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ff810f0b0 con 0x7f8ff8109aa0 2024-04-24T17:27:59.904 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.796+0000 7f8feffff700 1 -- 172.21.15.32:0/3369997358 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f8ff4001580 con 0x7f8ff8109aa0 2024-04-24T17:27:59.905 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.796+0000 7f8feffff700 1 -- 172.21.15.32:0/3369997358 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(25 keys) ==== 1042+0+0 (unknown 2768897130 0 0) 0x7f8ff4001c60 con 0x7f8ff8109aa0 2024-04-24T17:27:59.905 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.796+0000 7f8feffff700 1 -- 172.21.15.32:0/3369997358 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f8ff4005780 con 0x7f8ff8109aa0 2024-04-24T17:27:59.905 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.796+0000 7f8fff5bf700 1 -- 172.21.15.32:0/3369997358 >> v1:172.21.15.32:6789/0 conn(0x7f8ff8109aa0 legacy=0x7f8ff8109ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:59.905 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.796+0000 7f8fff5bf700 1 -- 172.21.15.32:0/3369997358 shutdown_connections 2024-04-24T17:27:59.905 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.796+0000 7f8fff5bf700 1 -- 172.21.15.32:0/3369997358 >> 172.21.15.32:0/3369997358 conn(0x7f8ff807ced0 msgr2=0x7f8ff81074a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:27:59.905 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.796+0000 7f8fff5bf700 1 -- 172.21.15.32:0/3369997358 shutdown_connections 2024-04-24T17:27:59.905 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.796+0000 7f8fff5bf700 1 -- 172.21.15.32:0/3369997358 wait complete. 2024-04-24T17:27:59.905 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.796+0000 7f8fff5bf700 1 librados: starting msgr at 2024-04-24T17:27:59.905 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.796+0000 7f8fff5bf700 1 librados: starting objecter 2024-04-24T17:27:59.905 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.796+0000 7f8fff5bf700 1 Processor -- start 2024-04-24T17:27:59.905 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.796+0000 7f8fff5bf700 1 -- start start 2024-04-24T17:27:59.905 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.796+0000 7f8fff5bf700 1 librados: setting wanted keys 2024-04-24T17:27:59.905 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.796+0000 7f8fff5bf700 1 librados: calling monclient init 2024-04-24T17:27:59.905 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.796+0000 7f8fff5bf700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8ff819e180 con 0x7f8ff819d650 2024-04-24T17:27:59.905 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.796+0000 7f8ffd35b700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f8ff819d650 0x7f8ff819da70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51542/0 (socket says 172.21.15.32:51542) 2024-04-24T17:27:59.906 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.796+0000 7f8ffd35b700 1 -- 172.21.15.32:0/2478026273 learned_addr learned my addr 172.21.15.32:0/2478026273 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:27:59.906 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.796+0000 7f8fee7fc700 10 client.?.objecter ms_handle_connect 0x7f8ff819d650 2024-04-24T17:27:59.906 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.796+0000 7f8fee7fc700 10 client.?.objecter resend_mon_ops 2024-04-24T17:27:59.906 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.796+0000 7f8fee7fc700 1 -- 172.21.15.32:0/2478026273 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2027480805 0 0) 0x7f8ff819e180 con 0x7f8ff819d650 2024-04-24T17:27:59.906 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.796+0000 7f8fee7fc700 1 -- 172.21.15.32:0/2478026273 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8fe40035b0 con 0x7f8ff819d650 2024-04-24T17:27:59.906 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.796+0000 7f8fee7fc700 1 -- 172.21.15.32:0/2478026273 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1679427071 0 0) 0x7f8fe40035b0 con 0x7f8ff819d650 2024-04-24T17:27:59.906 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.796+0000 7f8fee7fc700 1 -- 172.21.15.32:0/2478026273 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8ff819e180 con 0x7f8ff819d650 2024-04-24T17:27:59.906 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.800+0000 7f8fee7fc700 1 -- 172.21.15.32:0/2478026273 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f8ff4001080 con 0x7f8ff819d650 2024-04-24T17:27:59.906 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.800+0000 7f8fee7fc700 1 -- 172.21.15.32:0/2478026273 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 563381357 0 0) 0x7f8ff819e180 con 0x7f8ff819d650 2024-04-24T17:27:59.906 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.800+0000 7f8fee7fc700 1 -- 172.21.15.32:0/2478026273 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ff81a2b70 con 0x7f8ff819d650 2024-04-24T17:27:59.906 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.800+0000 7f8fff5bf700 1 -- 172.21.15.32:0/2478026273 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8ff81a1b60 con 0x7f8ff819d650 2024-04-24T17:27:59.906 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.800+0000 7f8fff5bf700 10 client.14144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-24T17:27:59.906 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.800+0000 7f8fff5bf700 1 -- 172.21.15.32:0/2478026273 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8ff81a20a0 con 0x7f8ff819d650 2024-04-24T17:27:59.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.800+0000 7f8fff5bf700 1 librados: init done 2024-04-24T17:27:59.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.800+0000 7f8fee7fc700 1 -- 172.21.15.32:0/2478026273 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(25 keys) ==== 1042+0+0 (unknown 2768897130 0 0) 0x7f8ff4001c20 con 0x7f8ff819d650 2024-04-24T17:27:59.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.800+0000 7f8fee7fc700 1 -- 172.21.15.32:0/2478026273 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f8ff4005fc0 con 0x7f8ff819d650 2024-04-24T17:27:59.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.800+0000 7f8fee7fc700 1 -- 172.21.15.32:0/2478026273 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 8) ==== 46523+0+0 (unknown 1814385894 0 0) 0x7f8ff40017a0 con 0x7f8ff819d650 2024-04-24T17:27:59.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.800+0000 7f8fff5bf700 1 -- 172.21.15.32:0/2478026273 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fdc005150 con 0x7f8ff819d650 2024-04-24T17:27:59.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.800+0000 7f8fee7fc700 1 -- 172.21.15.32:0/2478026273 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 3338368936 0 0) 0x7f8ff4048100 con 0x7f8ff819d650 2024-04-24T17:27:59.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.800+0000 7f8fee7fc700 10 client.14144.objecter ms_dispatch 0x7f8ff8001550 osd_map(2..2 src has 1..2) 2024-04-24T17:27:59.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.800+0000 7f8fee7fc700 3 client.14144.objecter handle_osd_map got epochs [2,2] > 0 2024-04-24T17:27:59.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.800+0000 7f8fee7fc700 3 client.14144.objecter handle_osd_map decoding full epoch 2 2024-04-24T17:27:59.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.800+0000 7f8fee7fc700 20 client.14144.objecter dump_active .. 0 homeless 2024-04-24T17:27:59.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.800+0000 7f8fee7fc700 10 client.14144.objecter ms_handle_connect 0x7f8fe403c370 2024-04-24T17:27:59.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:58.804+0000 7f8fee7fc700 1 -- 172.21.15.32:0/2478026273 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188728 (unknown 1092875540 0 2656817166) 0x7f8ff4016910 con 0x7f8ff819d650 2024-04-24T17:27:59.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:59.216+0000 7f8fff5bf700 1 -- 172.21.15.32:0/2478026273 --> v1:172.21.15.32:6800/3410850029 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f8fdc000bc0 con 0x7f8fe403c370 2024-04-24T17:27:59.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:59.228+0000 7f8fee7fc700 1 -- 172.21.15.32:0/2478026273 <== mgr.14120 v1:172.21.15.32:6800/3410850029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (unknown 0 0 325935098) 0x7f8fdc000bc0 con 0x7f8fe403c370 2024-04-24T17:27:59.907 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:59.228+0000 7f8fff5bf700 10 librados: watch_flush enter 2024-04-24T17:27:59.908 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:59.228+0000 7f8fff5bf700 10 librados: watch_flush exit 2024-04-24T17:27:59.908 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:59.228+0000 7f8fff5bf700 20 client.14144.objecter shutdown clearing up homeless session... 2024-04-24T17:27:59.908 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:59.228+0000 7f8fff5bf700 10 client.14144.objecter successfully canceled tick 2024-04-24T17:27:59.908 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:59.228+0000 7f8fff5bf700 1 -- 172.21.15.32:0/2478026273 >> v1:172.21.15.32:6800/3410850029 conn(0x7f8fe403c370 legacy=0x7f8fe403e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:59.908 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:59.228+0000 7f8fff5bf700 1 -- 172.21.15.32:0/2478026273 >> v1:172.21.15.32:6789/0 conn(0x7f8ff819d650 legacy=0x7f8ff819da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:27:59.908 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:59.228+0000 7f8fff5bf700 1 -- 172.21.15.32:0/2478026273 shutdown_connections 2024-04-24T17:27:59.908 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:59.228+0000 7f8fff5bf700 1 -- 172.21.15.32:0/2478026273 >> 172.21.15.32:0/2478026273 conn(0x7f8ff8109aa0 msgr2=0x7f8ff81074a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:27:59.908 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:59.228+0000 7f8fff5bf700 1 -- 172.21.15.32:0/2478026273 shutdown_connections 2024-04-24T17:27:59.908 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:59.228+0000 7f8fff5bf700 1 -- 172.21.15.32:0/2478026273 wait complete. 2024-04-24T17:27:59.908 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:27:59.228+0000 7f8fff5bf700 1 librados: shutdown 2024-04-24T17:28:00.501 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:00 smithi032 bash[21457]: cluster 2024-04-24T17:27:59.204615+0000 mgr.y (mgr.14120) 19 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:28:00.501 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:00 smithi032 bash[21457]: cluster 2024-04-24T17:27:59.204615+0000 mgr.y (mgr.14120) 19 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:28:00.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:00 smithi032 bash[21457]: audit 2024-04-24T17:27:59.222650+0000 mgr.y (mgr.14120) 20 : audit [DBG] from='client.14144 v1:172.21.15.32:0/2478026273' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T17:28:00.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:00 smithi032 bash[21457]: audit 2024-04-24T17:27:59.222650+0000 mgr.y (mgr.14120) 20 : audit [DBG] from='client.14144 v1:172.21.15.32:0/2478026273' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T17:28:00.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:00 smithi032 bash[21457]: cephadm 2024-04-24T17:27:59.225358+0000 mgr.y (mgr.14120) 21 : cephadm [INF] Saving service mgr spec with placement count:2 2024-04-24T17:28:00.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:00 smithi032 bash[21457]: cephadm 2024-04-24T17:27:59.225358+0000 mgr.y (mgr.14120) 21 : cephadm [INF] Saving service mgr spec with placement count:2 2024-04-24T17:28:00.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:00 smithi032 bash[21457]: audit 2024-04-24T17:27:59.230137+0000 mon.a (mon.0) 70 : audit [INF] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' 2024-04-24T17:28:00.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:00 smithi032 bash[21457]: audit 2024-04-24T17:27:59.230137+0000 mon.a (mon.0) 70 : audit [INF] from='mgr.14120 v1:172.21.15.32:0/707313880' entity='mgr.y' 2024-04-24T17:28:01.875 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.800+0000 7fdb45d62700 1 Processor -- start 2024-04-24T17:28:01.876 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.800+0000 7fdb45d62700 1 -- start start 2024-04-24T17:28:01.876 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.800+0000 7fdb45d62700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdb4010bcd0 con 0x7fdb401078a0 2024-04-24T17:28:01.876 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.800+0000 7fdb3f7fe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fdb401078a0 0x7fdb40107cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51556/0 (socket says 172.21.15.32:51556) 2024-04-24T17:28:01.876 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.800+0000 7fdb3f7fe700 1 -- 172.21.15.32:0/2893706361 learned_addr learned my addr 172.21.15.32:0/2893706361 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:28:01.876 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.800+0000 7fdb3e7fc700 1 -- 172.21.15.32:0/2893706361 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2766989686 0 0) 0x7fdb4010bcd0 con 0x7fdb401078a0 2024-04-24T17:28:01.876 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.800+0000 7fdb3e7fc700 1 -- 172.21.15.32:0/2893706361 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb280035b0 con 0x7fdb401078a0 2024-04-24T17:28:01.876 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.800+0000 7fdb3e7fc700 1 -- 172.21.15.32:0/2893706361 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4072675346 0 0) 0x7fdb280035b0 con 0x7fdb401078a0 2024-04-24T17:28:01.876 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.800+0000 7fdb3e7fc700 1 -- 172.21.15.32:0/2893706361 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb4010ceb0 con 0x7fdb401078a0 2024-04-24T17:28:01.876 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.800+0000 7fdb3e7fc700 1 -- 172.21.15.32:0/2893706361 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fdb30001580 con 0x7fdb401078a0 2024-04-24T17:28:01.876 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.800+0000 7fdb3e7fc700 1 -- 172.21.15.32:0/2893706361 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(25 keys) ==== 1042+0+0 (unknown 2768897130 0 0) 0x7fdb30001c60 con 0x7fdb401078a0 2024-04-24T17:28:01.876 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.800+0000 7fdb3e7fc700 1 -- 172.21.15.32:0/2893706361 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fdb30005780 con 0x7fdb401078a0 2024-04-24T17:28:01.876 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.800+0000 7fdb45d62700 1 -- 172.21.15.32:0/2893706361 >> v1:172.21.15.32:6789/0 conn(0x7fdb401078a0 legacy=0x7fdb40107cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:28:01.876 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.800+0000 7fdb45d62700 1 -- 172.21.15.32:0/2893706361 shutdown_connections 2024-04-24T17:28:01.876 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.800+0000 7fdb45d62700 1 -- 172.21.15.32:0/2893706361 >> 172.21.15.32:0/2893706361 conn(0x7fdb40102e40 msgr2=0x7fdb40105280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:28:01.877 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.800+0000 7fdb45d62700 1 -- 172.21.15.32:0/2893706361 shutdown_connections 2024-04-24T17:28:01.877 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.800+0000 7fdb45d62700 1 -- 172.21.15.32:0/2893706361 wait complete. 2024-04-24T17:28:01.877 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.800+0000 7fdb45d62700 1 librados: starting msgr at 2024-04-24T17:28:01.877 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.800+0000 7fdb45d62700 1 librados: starting objecter 2024-04-24T17:28:01.877 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.800+0000 7fdb45d62700 1 Processor -- start 2024-04-24T17:28:01.877 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.800+0000 7fdb45d62700 1 -- start start 2024-04-24T17:28:01.877 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.800+0000 7fdb45d62700 1 librados: setting wanted keys 2024-04-24T17:28:01.877 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.800+0000 7fdb45d62700 1 librados: calling monclient init 2024-04-24T17:28:01.877 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.800+0000 7fdb45d62700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdb4019ddf0 con 0x7fdb401078a0 2024-04-24T17:28:01.877 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.800+0000 7fdb3f7fe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fdb401078a0 0x7fdb4019d6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51568/0 (socket says 172.21.15.32:51568) 2024-04-24T17:28:01.877 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.800+0000 7fdb3f7fe700 1 -- 172.21.15.32:0/4191855618 learned_addr learned my addr 172.21.15.32:0/4191855618 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:28:01.877 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.800+0000 7fdb3cff9700 10 client.?.objecter ms_handle_connect 0x7fdb401078a0 2024-04-24T17:28:01.877 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.800+0000 7fdb3cff9700 10 client.?.objecter resend_mon_ops 2024-04-24T17:28:01.877 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.800+0000 7fdb3cff9700 1 -- 172.21.15.32:0/4191855618 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1412935542 0 0) 0x7fdb4019ddf0 con 0x7fdb401078a0 2024-04-24T17:28:01.877 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.800+0000 7fdb3cff9700 1 -- 172.21.15.32:0/4191855618 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb280035b0 con 0x7fdb401078a0 2024-04-24T17:28:01.878 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.804+0000 7fdb3cff9700 1 -- 172.21.15.32:0/4191855618 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3768126036 0 0) 0x7fdb280035b0 con 0x7fdb401078a0 2024-04-24T17:28:01.878 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.804+0000 7fdb3cff9700 1 -- 172.21.15.32:0/4191855618 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdb4019ddf0 con 0x7fdb401078a0 2024-04-24T17:28:01.878 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.804+0000 7fdb3cff9700 1 -- 172.21.15.32:0/4191855618 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fdb30001080 con 0x7fdb401078a0 2024-04-24T17:28:01.878 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.804+0000 7fdb3cff9700 1 -- 172.21.15.32:0/4191855618 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1574744983 0 0) 0x7fdb4019ddf0 con 0x7fdb401078a0 2024-04-24T17:28:01.878 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.804+0000 7fdb3cff9700 1 -- 172.21.15.32:0/4191855618 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb4019dfc0 con 0x7fdb401078a0 2024-04-24T17:28:01.878 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.804+0000 7fdb45d62700 1 -- 172.21.15.32:0/4191855618 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdb4019e2b0 con 0x7fdb401078a0 2024-04-24T17:28:01.878 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.804+0000 7fdb45d62700 10 client.14146.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-24T17:28:01.878 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.804+0000 7fdb45d62700 1 -- 172.21.15.32:0/4191855618 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdb401a1e00 con 0x7fdb401078a0 2024-04-24T17:28:01.878 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.804+0000 7fdb45d62700 1 librados: init done 2024-04-24T17:28:01.878 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.804+0000 7fdb3cff9700 1 -- 172.21.15.32:0/4191855618 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(25 keys) ==== 1042+0+0 (unknown 2768897130 0 0) 0x7fdb300056b0 con 0x7fdb401078a0 2024-04-24T17:28:01.878 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.804+0000 7fdb3cff9700 1 -- 172.21.15.32:0/4191855618 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fdb30006030 con 0x7fdb401078a0 2024-04-24T17:28:01.878 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.804+0000 7fdb3cff9700 1 -- 172.21.15.32:0/4191855618 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 8) ==== 46523+0+0 (unknown 1814385894 0 0) 0x7fdb30007300 con 0x7fdb401078a0 2024-04-24T17:28:01.878 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.804+0000 7fdb45d62700 1 -- 172.21.15.32:0/4191855618 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb2c005150 con 0x7fdb401078a0 2024-04-24T17:28:01.878 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.804+0000 7fdb3cff9700 1 -- 172.21.15.32:0/4191855618 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 3338368936 0 0) 0x7fdb300480d0 con 0x7fdb401078a0 2024-04-24T17:28:01.879 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.804+0000 7fdb3cff9700 10 client.14146.objecter ms_dispatch 0x7fdb40001550 osd_map(2..2 src has 1..2) 2024-04-24T17:28:01.879 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.804+0000 7fdb3cff9700 3 client.14146.objecter handle_osd_map got epochs [2,2] > 0 2024-04-24T17:28:01.879 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.804+0000 7fdb3cff9700 3 client.14146.objecter handle_osd_map decoding full epoch 2 2024-04-24T17:28:01.879 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.804+0000 7fdb3cff9700 20 client.14146.objecter dump_active .. 0 homeless 2024-04-24T17:28:01.879 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.804+0000 7fdb3cff9700 10 client.14146.objecter ms_handle_connect 0x7fdb2803c370 2024-04-24T17:28:01.879 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:00.808+0000 7fdb3cff9700 1 -- 172.21.15.32:0/4191855618 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188728 (unknown 1092875540 0 2656817166) 0x7fdb300176e0 con 0x7fdb401078a0 2024-04-24T17:28:01.879 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:01.184+0000 7fdb45d62700 1 -- 172.21.15.32:0/4191855618 --> v1:172.21.15.32:6789/0 -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7fdb2c005440 con 0x7fdb401078a0 2024-04-24T17:28:01.879 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:01.188+0000 7fdb3cff9700 1 -- 172.21.15.32:0/4191855618 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (unknown 2684808024 0 0) 0x7fdb30016fe0 con 0x7fdb401078a0 2024-04-24T17:28:01.879 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:01.192+0000 7fdb45d62700 10 librados: watch_flush enter 2024-04-24T17:28:01.879 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:01.192+0000 7fdb45d62700 10 librados: watch_flush exit 2024-04-24T17:28:01.879 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:01.192+0000 7fdb45d62700 20 client.14146.objecter shutdown clearing up homeless session... 2024-04-24T17:28:01.879 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:01.192+0000 7fdb45d62700 10 client.14146.objecter successfully canceled tick 2024-04-24T17:28:01.879 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:01.192+0000 7fdb45d62700 1 -- 172.21.15.32:0/4191855618 >> v1:172.21.15.32:6800/3410850029 conn(0x7fdb2803c370 legacy=0x7fdb2803e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:28:01.879 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:01.192+0000 7fdb45d62700 1 -- 172.21.15.32:0/4191855618 >> v1:172.21.15.32:6789/0 conn(0x7fdb401078a0 legacy=0x7fdb4019d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:28:01.879 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:01.192+0000 7fdb45d62700 1 -- 172.21.15.32:0/4191855618 shutdown_connections 2024-04-24T17:28:01.880 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:01.192+0000 7fdb45d62700 1 -- 172.21.15.32:0/4191855618 >> 172.21.15.32:0/4191855618 conn(0x7fdb40102e40 msgr2=0x7fdb40103270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:28:01.880 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:01.192+0000 7fdb45d62700 1 -- 172.21.15.32:0/4191855618 shutdown_connections 2024-04-24T17:28:01.880 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:01.192+0000 7fdb45d62700 1 -- 172.21.15.32:0/4191855618 wait complete. 2024-04-24T17:28:01.880 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:01.192+0000 7fdb45d62700 1 librados: shutdown 2024-04-24T17:28:02.501 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:02 smithi032 bash[21457]: audit 2024-04-24T17:28:01.192194+0000 mon.a (mon.0) 71 : audit [INF] from='client.? v1:172.21.15.32:0/4191855618' entity='client.admin' 2024-04-24T17:28:02.501 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:02 smithi032 bash[21457]: audit 2024-04-24T17:28:01.192194+0000 mon.a (mon.0) 71 : audit [INF] from='client.? v1:172.21.15.32:0/4191855618' entity='client.admin' 2024-04-24T17:28:02.501 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:02 smithi032 bash[21457]: cluster 2024-04-24T17:28:01.204877+0000 mgr.y (mgr.14120) 22 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:28:02.501 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:02 smithi032 bash[21457]: cluster 2024-04-24T17:28:01.204877+0000 mgr.y (mgr.14120) 22 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:28:03.747 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.760+0000 7f4627539700 1 Processor -- start 2024-04-24T17:28:03.747 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.760+0000 7f4627539700 1 -- start start 2024-04-24T17:28:03.747 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.764+0000 7f4627539700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f462010ded0 con 0x7f4620109aa0 2024-04-24T17:28:03.747 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.764+0000 7f46252d5700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f4620109aa0 0x7f4620109ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51576/0 (socket says 172.21.15.32:51576) 2024-04-24T17:28:03.747 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.764+0000 7f46252d5700 1 -- 172.21.15.32:0/70920826 learned_addr learned my addr 172.21.15.32:0/70920826 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:28:03.747 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.764+0000 7f4617fff700 1 -- 172.21.15.32:0/70920826 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3350291644 0 0) 0x7f462010ded0 con 0x7f4620109aa0 2024-04-24T17:28:03.747 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.764+0000 7f4617fff700 1 -- 172.21.15.32:0/70920826 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f460c0035b0 con 0x7f4620109aa0 2024-04-24T17:28:03.747 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.764+0000 7f4617fff700 1 -- 172.21.15.32:0/70920826 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4173856361 0 0) 0x7f460c0035b0 con 0x7f4620109aa0 2024-04-24T17:28:03.748 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.764+0000 7f4617fff700 1 -- 172.21.15.32:0/70920826 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f462010f0b0 con 0x7f4620109aa0 2024-04-24T17:28:03.748 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.764+0000 7f4617fff700 1 -- 172.21.15.32:0/70920826 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f461c001580 con 0x7f4620109aa0 2024-04-24T17:28:03.748 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.764+0000 7f4617fff700 1 -- 172.21.15.32:0/70920826 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(25 keys) ==== 1042+0+0 (unknown 2768897130 0 0) 0x7f461c001d30 con 0x7f4620109aa0 2024-04-24T17:28:03.748 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.764+0000 7f4617fff700 1 -- 172.21.15.32:0/70920826 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f461c005960 con 0x7f4620109aa0 2024-04-24T17:28:03.748 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.764+0000 7f4627539700 1 -- 172.21.15.32:0/70920826 >> v1:172.21.15.32:6789/0 conn(0x7f4620109aa0 legacy=0x7f4620109ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:28:03.748 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.764+0000 7f4627539700 1 -- 172.21.15.32:0/70920826 shutdown_connections 2024-04-24T17:28:03.748 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.764+0000 7f4627539700 1 -- 172.21.15.32:0/70920826 >> 172.21.15.32:0/70920826 conn(0x7f462007ced0 msgr2=0x7f46201074a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:28:03.748 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.764+0000 7f4627539700 1 -- 172.21.15.32:0/70920826 shutdown_connections 2024-04-24T17:28:03.748 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.764+0000 7f4627539700 1 -- 172.21.15.32:0/70920826 wait complete. 2024-04-24T17:28:03.748 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.764+0000 7f4627539700 1 librados: starting msgr at 2024-04-24T17:28:03.748 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.764+0000 7f4627539700 1 librados: starting objecter 2024-04-24T17:28:03.748 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.764+0000 7f4627539700 1 Processor -- start 2024-04-24T17:28:03.748 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.764+0000 7f4627539700 1 -- start start 2024-04-24T17:28:03.749 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.764+0000 7f4627539700 1 librados: setting wanted keys 2024-04-24T17:28:03.749 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.764+0000 7f4627539700 1 librados: calling monclient init 2024-04-24T17:28:03.749 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.764+0000 7f4627539700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f462019e150 con 0x7f462019d620 2024-04-24T17:28:03.749 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.764+0000 7f46252d5700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f462019d620 0x7f462019da40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51584/0 (socket says 172.21.15.32:51584) 2024-04-24T17:28:03.749 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.764+0000 7f46252d5700 1 -- 172.21.15.32:0/1722370541 learned_addr learned my addr 172.21.15.32:0/1722370541 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:28:03.749 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.764+0000 7f46167fc700 10 client.?.objecter ms_handle_connect 0x7f462019d620 2024-04-24T17:28:03.749 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.764+0000 7f46167fc700 10 client.?.objecter resend_mon_ops 2024-04-24T17:28:03.749 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.764+0000 7f46167fc700 1 -- 172.21.15.32:0/1722370541 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2081338550 0 0) 0x7f462019e150 con 0x7f462019d620 2024-04-24T17:28:03.749 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.764+0000 7f46167fc700 1 -- 172.21.15.32:0/1722370541 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f460c0056e0 con 0x7f462019d620 2024-04-24T17:28:03.749 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.768+0000 7f46167fc700 1 -- 172.21.15.32:0/1722370541 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 780061262 0 0) 0x7f462010f0b0 con 0x7f462019d620 2024-04-24T17:28:03.749 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.768+0000 7f46167fc700 1 -- 172.21.15.32:0/1722370541 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f462019e150 con 0x7f462019d620 2024-04-24T17:28:03.749 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.768+0000 7f46167fc700 1 -- 172.21.15.32:0/1722370541 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f461c001080 con 0x7f462019d620 2024-04-24T17:28:03.749 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.768+0000 7f46167fc700 1 -- 172.21.15.32:0/1722370541 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4190441414 0 0) 0x7f462019e150 con 0x7f462019d620 2024-04-24T17:28:03.749 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.768+0000 7f46167fc700 1 -- 172.21.15.32:0/1722370541 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46201a2b40 con 0x7f462019d620 2024-04-24T17:28:03.750 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.768+0000 7f4627539700 1 -- 172.21.15.32:0/1722370541 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f46201a1b30 con 0x7f462019d620 2024-04-24T17:28:03.750 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.768+0000 7f4627539700 10 client.14148.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-24T17:28:03.750 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.768+0000 7f4627539700 1 -- 172.21.15.32:0/1722370541 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f46201a2070 con 0x7f462019d620 2024-04-24T17:28:03.750 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.768+0000 7f4627539700 1 librados: init done 2024-04-24T17:28:03.750 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.768+0000 7f46167fc700 1 -- 172.21.15.32:0/1722370541 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(25 keys) ==== 1042+0+0 (unknown 2768897130 0 0) 0x7f461c001900 con 0x7f462019d620 2024-04-24T17:28:03.750 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.768+0000 7f46167fc700 1 -- 172.21.15.32:0/1722370541 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f461c005ff0 con 0x7f462019d620 2024-04-24T17:28:03.750 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.768+0000 7f46167fc700 1 -- 172.21.15.32:0/1722370541 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 8) ==== 46523+0+0 (unknown 1814385894 0 0) 0x7f461c011840 con 0x7f462019d620 2024-04-24T17:28:03.750 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.768+0000 7f4627539700 1 -- 172.21.15.32:0/1722370541 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4604005150 con 0x7f462019d620 2024-04-24T17:28:03.750 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.768+0000 7f46167fc700 1 -- 172.21.15.32:0/1722370541 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 3338368936 0 0) 0x7f461c0480c0 con 0x7f462019d620 2024-04-24T17:28:03.750 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.768+0000 7f46167fc700 10 client.14148.objecter ms_dispatch 0x7f4620001550 osd_map(2..2 src has 1..2) 2024-04-24T17:28:03.750 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.768+0000 7f46167fc700 3 client.14148.objecter handle_osd_map got epochs [2,2] > 0 2024-04-24T17:28:03.750 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.768+0000 7f46167fc700 3 client.14148.objecter handle_osd_map decoding full epoch 2 2024-04-24T17:28:03.750 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.768+0000 7f46167fc700 20 client.14148.objecter dump_active .. 0 homeless 2024-04-24T17:28:03.750 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.768+0000 7f46167fc700 10 client.14148.objecter ms_handle_connect 0x7f460c03c3c0 2024-04-24T17:28:03.751 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:02.772+0000 7f46167fc700 1 -- 172.21.15.32:0/1722370541 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188728 (unknown 1092875540 0 2656817166) 0x7f460c0056e0 con 0x7f462019d620 2024-04-24T17:28:03.751 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:03.152+0000 7f4627539700 1 -- 172.21.15.32:0/1722370541 --> v1:172.21.15.32:6789/0 -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f4604005440 con 0x7f462019d620 2024-04-24T17:28:03.751 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:03.156+0000 7f46167fc700 1 -- 172.21.15.32:0/1722370541 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (unknown 1496729260 0 0) 0x7f461c0176b0 con 0x7f462019d620 2024-04-24T17:28:03.751 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:03.160+0000 7f4627539700 10 librados: watch_flush enter 2024-04-24T17:28:03.751 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:03.160+0000 7f4627539700 10 librados: watch_flush exit 2024-04-24T17:28:03.751 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:03.160+0000 7f4627539700 20 client.14148.objecter shutdown clearing up homeless session... 2024-04-24T17:28:03.751 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:03.160+0000 7f4627539700 10 client.14148.objecter successfully canceled tick 2024-04-24T17:28:03.751 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:03.160+0000 7f4627539700 1 -- 172.21.15.32:0/1722370541 >> v1:172.21.15.32:6800/3410850029 conn(0x7f460c03c3c0 legacy=0x7f460c03e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:28:03.751 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:03.160+0000 7f4627539700 1 -- 172.21.15.32:0/1722370541 >> v1:172.21.15.32:6789/0 conn(0x7f462019d620 legacy=0x7f462019da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:28:03.751 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:03.160+0000 7f4627539700 1 -- 172.21.15.32:0/1722370541 shutdown_connections 2024-04-24T17:28:03.751 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:03.160+0000 7f4627539700 1 -- 172.21.15.32:0/1722370541 >> 172.21.15.32:0/1722370541 conn(0x7f4620109aa0 msgr2=0x7f46201074a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:28:03.751 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:03.160+0000 7f4627539700 1 -- 172.21.15.32:0/1722370541 shutdown_connections 2024-04-24T17:28:03.751 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:03.160+0000 7f4627539700 1 -- 172.21.15.32:0/1722370541 wait complete. 2024-04-24T17:28:03.751 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:03.160+0000 7f4627539700 1 librados: shutdown 2024-04-24T17:28:03.752 INFO:teuthology.orchestra.run.smithi032.stdout:Enabling the dashboard module... 2024-04-24T17:28:04.501 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:04 smithi032 bash[21457]: audit 2024-04-24T17:28:03.161243+0000 mon.a (mon.0) 72 : audit [INF] from='client.? v1:172.21.15.32:0/1722370541' entity='client.admin' 2024-04-24T17:28:04.501 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:04 smithi032 bash[21457]: audit 2024-04-24T17:28:03.161243+0000 mon.a (mon.0) 72 : audit [INF] from='client.? v1:172.21.15.32:0/1722370541' entity='client.admin' 2024-04-24T17:28:04.501 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:04 smithi032 bash[21457]: cluster 2024-04-24T17:28:03.205137+0000 mgr.y (mgr.14120) 23 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:28:04.501 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:04 smithi032 bash[21457]: cluster 2024-04-24T17:28:03.205137+0000 mgr.y (mgr.14120) 23 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:28:05.751 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:05 smithi032 bash[21457]: audit 2024-04-24T17:28:05.239494+0000 mon.a (mon.0) 73 : audit [INF] from='client.? v1:172.21.15.32:0/197002970' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-24T17:28:05.752 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:05 smithi032 bash[21457]: audit 2024-04-24T17:28:05.239494+0000 mon.a (mon.0) 73 : audit [INF] from='client.? v1:172.21.15.32:0/197002970' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-24T17:28:05.752 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:28:05 smithi032 bash[21712]: ignoring --setuser ceph since I am not root 2024-04-24T17:28:05.752 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:28:05 smithi032 bash[21712]: ignoring --setgroup ceph since I am not root 2024-04-24T17:28:05.752 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:28:05 smithi032 bash[21712]: debug 2024-04-24T17:28:05.488+0000 7fd40e324200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T17:28:06.010 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.720+0000 7fb2d07d9700 1 Processor -- start 2024-04-24T17:28:06.010 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.720+0000 7fb2d07d9700 1 -- start start 2024-04-24T17:28:06.010 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.720+0000 7fb2d07d9700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb2c807ae20 con 0x7fb2c8108390 2024-04-24T17:28:06.010 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.720+0000 7fb2ce575700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb2c8108390 0x7fb2c810a780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51600/0 (socket says 172.21.15.32:51600) 2024-04-24T17:28:06.011 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.720+0000 7fb2ce575700 1 -- 172.21.15.32:0/925975112 learned_addr learned my addr 172.21.15.32:0/925975112 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:28:06.011 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.720+0000 7fb2cd573700 1 -- 172.21.15.32:0/925975112 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3337341689 0 0) 0x7fb2c807ae20 con 0x7fb2c8108390 2024-04-24T17:28:06.011 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.720+0000 7fb2cd573700 1 -- 172.21.15.32:0/925975112 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb2b40035b0 con 0x7fb2c8108390 2024-04-24T17:28:06.011 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.720+0000 7fb2cd573700 1 -- 172.21.15.32:0/925975112 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1388515368 0 0) 0x7fb2b40035b0 con 0x7fb2c8108390 2024-04-24T17:28:06.011 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.720+0000 7fb2cd573700 1 -- 172.21.15.32:0/925975112 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2c810ceb0 con 0x7fb2c8108390 2024-04-24T17:28:06.011 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.720+0000 7fb2cd573700 1 -- 172.21.15.32:0/925975112 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fb2c4001580 con 0x7fb2c8108390 2024-04-24T17:28:06.011 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.720+0000 7fb2cd573700 1 -- 172.21.15.32:0/925975112 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(25 keys) ==== 1042+0+0 (unknown 2768897130 0 0) 0x7fb2c4001c60 con 0x7fb2c8108390 2024-04-24T17:28:06.011 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.720+0000 7fb2cd573700 1 -- 172.21.15.32:0/925975112 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fb2c4005960 con 0x7fb2c8108390 2024-04-24T17:28:06.011 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.720+0000 7fb2d07d9700 1 -- 172.21.15.32:0/925975112 >> v1:172.21.15.32:6789/0 conn(0x7fb2c8108390 legacy=0x7fb2c810a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:28:06.011 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.720+0000 7fb2d07d9700 1 -- 172.21.15.32:0/925975112 shutdown_connections 2024-04-24T17:28:06.011 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.720+0000 7fb2d07d9700 1 -- 172.21.15.32:0/925975112 >> 172.21.15.32:0/925975112 conn(0x7fb2c8101d60 msgr2=0x7fb2c81041a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:28:06.011 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.720+0000 7fb2d07d9700 1 -- 172.21.15.32:0/925975112 shutdown_connections 2024-04-24T17:28:06.011 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.720+0000 7fb2d07d9700 1 -- 172.21.15.32:0/925975112 wait complete. 2024-04-24T17:28:06.011 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.720+0000 7fb2d07d9700 1 librados: starting msgr at 2024-04-24T17:28:06.012 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.720+0000 7fb2d07d9700 1 librados: starting objecter 2024-04-24T17:28:06.012 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.720+0000 7fb2d07d9700 1 Processor -- start 2024-04-24T17:28:06.012 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.720+0000 7fb2d07d9700 1 -- start start 2024-04-24T17:28:06.012 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.720+0000 7fb2d07d9700 1 librados: setting wanted keys 2024-04-24T17:28:06.012 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.720+0000 7fb2d07d9700 1 librados: calling monclient init 2024-04-24T17:28:06.012 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.724+0000 7fb2d07d9700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb2c819bc40 con 0x7fb2c8108390 2024-04-24T17:28:06.012 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.724+0000 7fb2ce575700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb2c8108390 0x7fb2c819b530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51604/0 (socket says 172.21.15.32:51604) 2024-04-24T17:28:06.012 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.724+0000 7fb2ce575700 1 -- 172.21.15.32:0/197002970 learned_addr learned my addr 172.21.15.32:0/197002970 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:28:06.012 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.724+0000 7fb2bf7fe700 10 client.?.objecter ms_handle_connect 0x7fb2c8108390 2024-04-24T17:28:06.012 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.724+0000 7fb2bf7fe700 10 client.?.objecter resend_mon_ops 2024-04-24T17:28:06.012 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.724+0000 7fb2bf7fe700 1 -- 172.21.15.32:0/197002970 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1787744521 0 0) 0x7fb2c819bc40 con 0x7fb2c8108390 2024-04-24T17:28:06.012 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.724+0000 7fb2bf7fe700 1 -- 172.21.15.32:0/197002970 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb2b40035b0 con 0x7fb2c8108390 2024-04-24T17:28:06.012 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.724+0000 7fb2bf7fe700 1 -- 172.21.15.32:0/197002970 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2609609380 0 0) 0x7fb2b40035b0 con 0x7fb2c8108390 2024-04-24T17:28:06.012 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.724+0000 7fb2bf7fe700 1 -- 172.21.15.32:0/197002970 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb2c819bc40 con 0x7fb2c8108390 2024-04-24T17:28:06.012 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.724+0000 7fb2bf7fe700 1 -- 172.21.15.32:0/197002970 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fb2c4001080 con 0x7fb2c8108390 2024-04-24T17:28:06.013 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.724+0000 7fb2bf7fe700 1 -- 172.21.15.32:0/197002970 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1909831320 0 0) 0x7fb2c819bc40 con 0x7fb2c8108390 2024-04-24T17:28:06.013 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.724+0000 7fb2bf7fe700 1 -- 172.21.15.32:0/197002970 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2c819be10 con 0x7fb2c8108390 2024-04-24T17:28:06.013 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.724+0000 7fb2d07d9700 1 -- 172.21.15.32:0/197002970 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb2c819c100 con 0x7fb2c8108390 2024-04-24T17:28:06.013 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.724+0000 7fb2d07d9700 10 client.14150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-24T17:28:06.013 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.724+0000 7fb2d07d9700 1 -- 172.21.15.32:0/197002970 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb2c819fc50 con 0x7fb2c8108390 2024-04-24T17:28:06.013 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.724+0000 7fb2d07d9700 1 librados: init done 2024-04-24T17:28:06.013 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.724+0000 7fb2bf7fe700 1 -- 172.21.15.32:0/197002970 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(25 keys) ==== 1042+0+0 (unknown 2768897130 0 0) 0x7fb2c40056e0 con 0x7fb2c8108390 2024-04-24T17:28:06.013 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.724+0000 7fb2bf7fe700 1 -- 172.21.15.32:0/197002970 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fb2c40060b0 con 0x7fb2c8108390 2024-04-24T17:28:06.013 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.724+0000 7fb2bf7fe700 1 -- 172.21.15.32:0/197002970 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 8) ==== 46523+0+0 (unknown 1814385894 0 0) 0x7fb2c4011820 con 0x7fb2c8108390 2024-04-24T17:28:06.013 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.724+0000 7fb2bf7fe700 1 -- 172.21.15.32:0/197002970 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 3338368936 0 0) 0x7fb2c4048000 con 0x7fb2c8108390 2024-04-24T17:28:06.013 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.724+0000 7fb2bf7fe700 10 client.14150.objecter ms_dispatch 0x7fb2c8001550 osd_map(2..2 src has 1..2) 2024-04-24T17:28:06.013 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.724+0000 7fb2bf7fe700 3 client.14150.objecter handle_osd_map got epochs [2,2] > 0 2024-04-24T17:28:06.013 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.724+0000 7fb2bf7fe700 3 client.14150.objecter handle_osd_map decoding full epoch 2 2024-04-24T17:28:06.013 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.724+0000 7fb2bf7fe700 20 client.14150.objecter dump_active .. 0 homeless 2024-04-24T17:28:06.014 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.724+0000 7fb2d07d9700 1 -- 172.21.15.32:0/197002970 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2ac005120 con 0x7fb2c8108390 2024-04-24T17:28:06.014 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.724+0000 7fb2bf7fe700 10 client.14150.objecter ms_handle_connect 0x7fb2b403c3a0 2024-04-24T17:28:06.014 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:04.728+0000 7fb2bf7fe700 1 -- 172.21.15.32:0/197002970 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188728 (unknown 1092875540 0 2656817166) 0x7fb2c4017730 con 0x7fb2c8108390 2024-04-24T17:28:06.014 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:05.236+0000 7fb2d07d9700 1 -- 172.21.15.32:0/197002970 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7fb2ac005410 con 0x7fb2c8108390 2024-04-24T17:28:06.014 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:05.288+0000 7fb2bf7fe700 1 -- 172.21.15.32:0/197002970 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (unknown 1498667528 0 0) 0x7fb2c4017030 con 0x7fb2c8108390 2024-04-24T17:28:06.014 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:05.292+0000 7fb2bf7fe700 1 -- 172.21.15.32:0/197002970 <== mon.0 v1:172.21.15.32:6789/0 11 ==== mgrmap(e 9) ==== 46536+0+0 (unknown 3295546111 0 0) 0x7fb2c4013670 con 0x7fb2c8108390 2024-04-24T17:28:06.014 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:05.292+0000 7fb2d07d9700 10 librados: watch_flush enter 2024-04-24T17:28:06.014 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:05.292+0000 7fb2d07d9700 10 librados: watch_flush exit 2024-04-24T17:28:06.014 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:05.292+0000 7fb2d07d9700 20 client.14150.objecter shutdown clearing up homeless session... 2024-04-24T17:28:06.014 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:05.292+0000 7fb2d07d9700 10 client.14150.objecter successfully canceled tick 2024-04-24T17:28:06.014 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:05.292+0000 7fb2d07d9700 1 -- 172.21.15.32:0/197002970 >> v1:172.21.15.32:6800/3410850029 conn(0x7fb2b403c3a0 legacy=0x7fb2b403e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:28:06.014 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:05.292+0000 7fb2d07d9700 1 -- 172.21.15.32:0/197002970 >> v1:172.21.15.32:6789/0 conn(0x7fb2c8108390 legacy=0x7fb2c819b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:28:06.014 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:05.292+0000 7fb2d07d9700 1 -- 172.21.15.32:0/197002970 shutdown_connections 2024-04-24T17:28:06.015 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:05.292+0000 7fb2d07d9700 1 -- 172.21.15.32:0/197002970 >> 172.21.15.32:0/197002970 conn(0x7fb2c8101d60 msgr2=0x7fb2c81921b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:28:06.015 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:05.292+0000 7fb2d07d9700 1 -- 172.21.15.32:0/197002970 shutdown_connections 2024-04-24T17:28:06.015 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:05.292+0000 7fb2d07d9700 1 -- 172.21.15.32:0/197002970 wait complete. 2024-04-24T17:28:06.015 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:05.292+0000 7fb2d07d9700 1 librados: shutdown 2024-04-24T17:28:06.657 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:06 smithi032 bash[21457]: audit 2024-04-24T17:28:05.291400+0000 mon.a (mon.0) 74 : audit [INF] from='client.? v1:172.21.15.32:0/197002970' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-24T17:28:06.657 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:06 smithi032 bash[21457]: audit 2024-04-24T17:28:05.291400+0000 mon.a (mon.0) 74 : audit [INF] from='client.? v1:172.21.15.32:0/197002970' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-24T17:28:06.657 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:06 smithi032 bash[21457]: cluster 2024-04-24T17:28:05.297308+0000 mon.a (mon.0) 75 : cluster [DBG] mgrmap e9: y(active, since 28s) 2024-04-24T17:28:06.657 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:06 smithi032 bash[21457]: cluster 2024-04-24T17:28:05.297308+0000 mon.a (mon.0) 75 : cluster [DBG] mgrmap e9: y(active, since 28s) 2024-04-24T17:28:07.501 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:28:07 smithi032 bash[21712]: debug 2024-04-24T17:28:07.176+0000 7fd40e324200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T17:28:07.501 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:28:07 smithi032 bash[21712]: debug 2024-04-24T17:28:07.276+0000 7fd40e324200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T17:28:07.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:07 smithi032 bash[21457]: audit 2024-04-24T17:28:07.349908+0000 mon.a (mon.0) 76 : audit [DBG] from='client.? v1:172.21.15.32:0/3847162637' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-24T17:28:07.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:07 smithi032 bash[21457]: audit 2024-04-24T17:28:07.349908+0000 mon.a (mon.0) 76 : audit [DBG] from='client.? v1:172.21.15.32:0/3847162637' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-24T17:28:08.001 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:28:07 smithi032 bash[21712]: debug 2024-04-24T17:28:07.536+0000 7fd40e324200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T17:28:08.084 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-04-24T17:28:08.084 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-24T17:28:08.084 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-24T17:28:08.084 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-04-24T17:28:08.084 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-24T17:28:08.084 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-04-24T17:28:08.084 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.836+0000 7fe6caa01700 1 Processor -- start 2024-04-24T17:28:08.084 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.836+0000 7fe6caa01700 1 -- start start 2024-04-24T17:28:08.084 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.836+0000 7fe6caa01700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe6c410aba0 con 0x7fe6c4106770 2024-04-24T17:28:08.084 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.836+0000 7fe6c3fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe6c4106770 0x7fe6c4106b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:43392/0 (socket says 172.21.15.32:43392) 2024-04-24T17:28:08.084 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.836+0000 7fe6c3fff700 1 -- 172.21.15.32:0/1735066204 learned_addr learned my addr 172.21.15.32:0/1735066204 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:28:08.084 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.836+0000 7fe6c2ffd700 1 -- 172.21.15.32:0/1735066204 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 394480278 0 0) 0x7fe6c410aba0 con 0x7fe6c4106770 2024-04-24T17:28:08.084 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.836+0000 7fe6c2ffd700 1 -- 172.21.15.32:0/1735066204 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6b00035b0 con 0x7fe6c4106770 2024-04-24T17:28:08.085 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.836+0000 7fe6c2ffd700 1 -- 172.21.15.32:0/1735066204 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1787111241 0 0) 0x7fe6b00035b0 con 0x7fe6c4106770 2024-04-24T17:28:08.085 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.836+0000 7fe6c2ffd700 1 -- 172.21.15.32:0/1735066204 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6c410bd80 con 0x7fe6c4106770 2024-04-24T17:28:08.085 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.836+0000 7fe6c2ffd700 1 -- 172.21.15.32:0/1735066204 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fe6b4001580 con 0x7fe6c4106770 2024-04-24T17:28:08.085 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.836+0000 7fe6c2ffd700 1 -- 172.21.15.32:0/1735066204 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(25 keys) ==== 1042+0+0 (unknown 2768897130 0 0) 0x7fe6b4001c60 con 0x7fe6c4106770 2024-04-24T17:28:08.085 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.836+0000 7fe6c2ffd700 1 -- 172.21.15.32:0/1735066204 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fe6b4005780 con 0x7fe6c4106770 2024-04-24T17:28:08.085 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.836+0000 7fe6caa01700 1 -- 172.21.15.32:0/1735066204 >> v1:172.21.15.32:6789/0 conn(0x7fe6c4106770 legacy=0x7fe6c4106b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:28:08.085 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.836+0000 7fe6caa01700 1 -- 172.21.15.32:0/1735066204 shutdown_connections 2024-04-24T17:28:08.085 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.836+0000 7fe6caa01700 1 -- 172.21.15.32:0/1735066204 >> 172.21.15.32:0/1735066204 conn(0x7fe6c4101cd0 msgr2=0x7fe6c4104150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:28:08.085 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.836+0000 7fe6caa01700 1 -- 172.21.15.32:0/1735066204 shutdown_connections 2024-04-24T17:28:08.085 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.836+0000 7fe6caa01700 1 -- 172.21.15.32:0/1735066204 wait complete. 2024-04-24T17:28:08.085 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.840+0000 7fe6caa01700 1 librados: starting msgr at 2024-04-24T17:28:08.085 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.840+0000 7fe6caa01700 1 librados: starting objecter 2024-04-24T17:28:08.085 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.840+0000 7fe6caa01700 1 Processor -- start 2024-04-24T17:28:08.085 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.840+0000 7fe6caa01700 1 -- start start 2024-04-24T17:28:08.085 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.840+0000 7fe6caa01700 1 librados: setting wanted keys 2024-04-24T17:28:08.086 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.840+0000 7fe6caa01700 1 librados: calling monclient init 2024-04-24T17:28:08.086 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.840+0000 7fe6caa01700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe6c419de00 con 0x7fe6c4106770 2024-04-24T17:28:08.086 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.840+0000 7fe6c3fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe6c4106770 0x7fe6c419d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:43406/0 (socket says 172.21.15.32:43406) 2024-04-24T17:28:08.086 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.840+0000 7fe6c3fff700 1 -- 172.21.15.32:0/3847162637 learned_addr learned my addr 172.21.15.32:0/3847162637 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:28:08.086 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.840+0000 7fe6c17fa700 10 client.?.objecter ms_handle_connect 0x7fe6c4106770 2024-04-24T17:28:08.086 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.840+0000 7fe6c17fa700 10 client.?.objecter resend_mon_ops 2024-04-24T17:28:08.086 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.840+0000 7fe6c17fa700 1 -- 172.21.15.32:0/3847162637 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3855896980 0 0) 0x7fe6c419de00 con 0x7fe6c4106770 2024-04-24T17:28:08.086 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.840+0000 7fe6c17fa700 1 -- 172.21.15.32:0/3847162637 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6b00035b0 con 0x7fe6c4106770 2024-04-24T17:28:08.086 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.840+0000 7fe6c17fa700 1 -- 172.21.15.32:0/3847162637 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 281843218 0 0) 0x7fe6b00035b0 con 0x7fe6c4106770 2024-04-24T17:28:08.086 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.840+0000 7fe6c17fa700 1 -- 172.21.15.32:0/3847162637 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe6c419de00 con 0x7fe6c4106770 2024-04-24T17:28:08.086 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.840+0000 7fe6c17fa700 1 -- 172.21.15.32:0/3847162637 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fe6b4001080 con 0x7fe6c4106770 2024-04-24T17:28:08.086 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.840+0000 7fe6c17fa700 1 -- 172.21.15.32:0/3847162637 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4032499218 0 0) 0x7fe6c419de00 con 0x7fe6c4106770 2024-04-24T17:28:08.086 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.840+0000 7fe6c17fa700 1 -- 172.21.15.32:0/3847162637 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6c419dfd0 con 0x7fe6c4106770 2024-04-24T17:28:08.086 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.840+0000 7fe6caa01700 1 -- 172.21.15.32:0/3847162637 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe6c419e2c0 con 0x7fe6c4106770 2024-04-24T17:28:08.087 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.840+0000 7fe6caa01700 10 client.14154.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-24T17:28:08.087 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.840+0000 7fe6caa01700 1 -- 172.21.15.32:0/3847162637 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe6c41a1e10 con 0x7fe6c4106770 2024-04-24T17:28:08.087 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.840+0000 7fe6caa01700 1 librados: init done 2024-04-24T17:28:08.087 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.840+0000 7fe6c17fa700 1 -- 172.21.15.32:0/3847162637 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(25 keys) ==== 1042+0+0 (unknown 2768897130 0 0) 0x7fe6b40049d0 con 0x7fe6c4106770 2024-04-24T17:28:08.087 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.840+0000 7fe6c17fa700 1 -- 172.21.15.32:0/3847162637 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fe6b4005f20 con 0x7fe6c4106770 2024-04-24T17:28:08.087 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.840+0000 7fe6c17fa700 1 -- 172.21.15.32:0/3847162637 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 9) ==== 46536+0+0 (unknown 3295546111 0 0) 0x7fe6b4011770 con 0x7fe6c4106770 2024-04-24T17:28:08.087 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.840+0000 7fe6caa01700 1 -- 172.21.15.32:0/3847162637 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6a4005150 con 0x7fe6c4106770 2024-04-24T17:28:08.087 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.840+0000 7fe6c17fa700 1 -- 172.21.15.32:0/3847162637 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 3338368936 0 0) 0x7fe6b4048e90 con 0x7fe6c4106770 2024-04-24T17:28:08.087 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.840+0000 7fe6c17fa700 10 client.14154.objecter ms_dispatch 0x7fe6c4001550 osd_map(2..2 src has 1..2) 2024-04-24T17:28:08.087 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.840+0000 7fe6c17fa700 3 client.14154.objecter handle_osd_map got epochs [2,2] > 0 2024-04-24T17:28:08.087 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.840+0000 7fe6c17fa700 3 client.14154.objecter handle_osd_map decoding full epoch 2 2024-04-24T17:28:08.087 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.840+0000 7fe6c37fe700 1 -- 172.21.15.32:0/3847162637 >> v1:172.21.15.32:6800/3410850029 conn(0x7fe6b003c3c0 legacy=0x7fe6b003e880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.32:6800/3410850029 2024-04-24T17:28:08.087 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.840+0000 7fe6c17fa700 20 client.14154.objecter dump_active .. 0 homeless 2024-04-24T17:28:08.087 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:06.848+0000 7fe6c17fa700 1 -- 172.21.15.32:0/3847162637 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188728 (unknown 1092875540 0 2656817166) 0x7fe6b4017700 con 0x7fe6c4106770 2024-04-24T17:28:08.088 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:07.044+0000 7fe6c37fe700 1 -- 172.21.15.32:0/3847162637 >> v1:172.21.15.32:6800/3410850029 conn(0x7fe6b003c3c0 legacy=0x7fe6b003e880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.32:6800/3410850029 2024-04-24T17:28:08.088 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:07.344+0000 7fe6caa01700 1 -- 172.21.15.32:0/3847162637 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fe6a4005c50 con 0x7fe6c4106770 2024-04-24T17:28:08.088 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:07.344+0000 7fe6c17fa700 1 -- 172.21.15.32:0/3847162637 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (unknown 2005831594 0 1748205903) 0x7fe6b4017000 con 0x7fe6c4106770 2024-04-24T17:28:08.088 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:07.348+0000 7fe6caa01700 10 librados: watch_flush enter 2024-04-24T17:28:08.088 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:07.348+0000 7fe6caa01700 10 librados: watch_flush exit 2024-04-24T17:28:08.088 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:07.348+0000 7fe6caa01700 20 client.14154.objecter shutdown clearing up homeless session... 2024-04-24T17:28:08.088 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:07.348+0000 7fe6caa01700 10 client.14154.objecter successfully canceled tick 2024-04-24T17:28:08.088 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:07.348+0000 7fe6caa01700 1 -- 172.21.15.32:0/3847162637 >> v1:172.21.15.32:6800/3410850029 conn(0x7fe6b003c3c0 legacy=0x7fe6b003e880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T17:28:08.088 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:07.348+0000 7fe6caa01700 1 -- 172.21.15.32:0/3847162637 >> v1:172.21.15.32:6789/0 conn(0x7fe6c4106770 legacy=0x7fe6c419d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:28:08.088 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:07.348+0000 7fe6caa01700 1 -- 172.21.15.32:0/3847162637 shutdown_connections 2024-04-24T17:28:08.088 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:07.348+0000 7fe6caa01700 1 -- 172.21.15.32:0/3847162637 >> 172.21.15.32:0/3847162637 conn(0x7fe6c4101cd0 msgr2=0x7fe6c4104150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:28:08.088 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:07.348+0000 7fe6caa01700 1 -- 172.21.15.32:0/3847162637 shutdown_connections 2024-04-24T17:28:08.088 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:07.348+0000 7fe6caa01700 1 -- 172.21.15.32:0/3847162637 wait complete. 2024-04-24T17:28:08.088 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:07.348+0000 7fe6caa01700 1 librados: shutdown 2024-04-24T17:28:08.088 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for the mgr to restart... 2024-04-24T17:28:08.088 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for mgr epoch 9... 2024-04-24T17:28:08.751 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:28:08 smithi032 bash[21712]: debug 2024-04-24T17:28:08.372+0000 7fd40e324200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T17:28:08.752 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:28:08 smithi032 bash[21712]: debug 2024-04-24T17:28:08.488+0000 7fd40e324200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T17:28:08.752 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:28:08 smithi032 bash[21712]: debug 2024-04-24T17:28:08.596+0000 7fd40e324200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T17:28:09.093 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:28:08 smithi032 bash[21712]: debug 2024-04-24T17:28:08.972+0000 7fd40e324200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-24T17:28:09.501 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:28:09 smithi032 bash[21712]: debug 2024-04-24T17:28:09.092+0000 7fd40e324200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T17:28:09.501 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:28:09 smithi032 bash[21712]: debug 2024-04-24T17:28:09.200+0000 7fd40e324200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T17:28:10.001 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:28:09 smithi032 bash[21712]: debug 2024-04-24T17:28:09.572+0000 7fd40e324200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T17:28:10.002 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:28:09 smithi032 bash[21712]: debug 2024-04-24T17:28:09.816+0000 7fd40e324200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T17:28:10.499 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:28:10 smithi032 bash[21712]: debug 2024-04-24T17:28:10.092+0000 7fd40e324200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T17:28:10.499 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:28:10 smithi032 bash[21712]: debug 2024-04-24T17:28:10.212+0000 7fd40e324200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T17:28:10.751 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:28:10 smithi032 bash[21712]: debug 2024-04-24T17:28:10.496+0000 7fd40e324200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T17:28:11.251 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:28:10 smithi032 bash[21712]: debug 2024-04-24T17:28:10.920+0000 7fd40e324200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T17:28:11.252 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:28:11 smithi032 bash[21712]: debug 2024-04-24T17:28:11.168+0000 7fd40e324200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T17:28:11.558 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:28:11 smithi032 bash[21712]: debug 2024-04-24T17:28:11.296+0000 7fd40e324200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T17:28:12.001 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:28:11 smithi032 bash[21712]: debug 2024-04-24T17:28:11.556+0000 7fd40e324200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T17:28:13.001 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:28:12 smithi032 bash[21712]: debug 2024-04-24T17:28:12.600+0000 7fd40e324200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T17:28:13.002 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:28:12 smithi032 bash[21712]: debug 2024-04-24T17:28:12.720+0000 7fd40e324200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T17:28:13.501 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:28:13 smithi032 bash[21712]: debug 2024-04-24T17:28:13.028+0000 7fd40e324200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T17:28:13.502 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:28:13 smithi032 bash[21712]: debug 2024-04-24T17:28:13.148+0000 7fd40e324200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T17:28:13.880 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:28:13 smithi032 bash[21712]: debug 2024-04-24T17:28:13.624+0000 7fd40e324200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T17:28:13.880 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:28:13 smithi032 bash[21712]: debug 2024-04-24T17:28:13.748+0000 7fd40e324200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T17:28:13.880 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:13 smithi032 bash[21457]: cluster 2024-04-24T17:28:13.773497+0000 mon.a (mon.0) 77 : cluster [INF] Active manager daemon y restarted 2024-04-24T17:28:13.880 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:13 smithi032 bash[21457]: cluster 2024-04-24T17:28:13.773497+0000 mon.a (mon.0) 77 : cluster [INF] Active manager daemon y restarted 2024-04-24T17:28:13.880 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:13 smithi032 bash[21457]: cluster 2024-04-24T17:28:13.773941+0000 mon.a (mon.0) 78 : cluster [INF] Activating manager daemon y 2024-04-24T17:28:13.880 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:13 smithi032 bash[21457]: cluster 2024-04-24T17:28:13.773941+0000 mon.a (mon.0) 78 : cluster [INF] Activating manager daemon y 2024-04-24T17:28:13.880 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:13 smithi032 bash[21457]: cluster 2024-04-24T17:28:13.780644+0000 mon.a (mon.0) 79 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-04-24T17:28:13.880 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:13 smithi032 bash[21457]: cluster 2024-04-24T17:28:13.780644+0000 mon.a (mon.0) 79 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-04-24T17:28:13.881 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:13 smithi032 bash[21457]: cluster 2024-04-24T17:28:13.780921+0000 mon.a (mon.0) 80 : cluster [DBG] mgrmap e10: y(active, starting, since 0.00713089s) 2024-04-24T17:28:13.881 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:13 smithi032 bash[21457]: cluster 2024-04-24T17:28:13.780921+0000 mon.a (mon.0) 80 : cluster [DBG] mgrmap e10: y(active, starting, since 0.00713089s) 2024-04-24T17:28:13.881 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:13 smithi032 bash[21457]: audit 2024-04-24T17:28:13.782467+0000 mon.a (mon.0) 81 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T17:28:13.881 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:13 smithi032 bash[21457]: audit 2024-04-24T17:28:13.782467+0000 mon.a (mon.0) 81 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T17:28:13.881 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:13 smithi032 bash[21457]: audit 2024-04-24T17:28:13.784318+0000 mon.a (mon.0) 82 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-24T17:28:13.881 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:13 smithi032 bash[21457]: audit 2024-04-24T17:28:13.784318+0000 mon.a (mon.0) 82 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-24T17:28:13.881 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:13 smithi032 bash[21457]: audit 2024-04-24T17:28:13.785246+0000 mon.a (mon.0) 83 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-24T17:28:13.881 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:13 smithi032 bash[21457]: audit 2024-04-24T17:28:13.785246+0000 mon.a (mon.0) 83 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-24T17:28:13.881 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:13 smithi032 bash[21457]: audit 2024-04-24T17:28:13.785404+0000 mon.a (mon.0) 84 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-24T17:28:13.881 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:13 smithi032 bash[21457]: audit 2024-04-24T17:28:13.785404+0000 mon.a (mon.0) 84 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-24T17:28:13.881 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:13 smithi032 bash[21457]: audit 2024-04-24T17:28:13.785524+0000 mon.a (mon.0) 85 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-24T17:28:13.881 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:13 smithi032 bash[21457]: audit 2024-04-24T17:28:13.785524+0000 mon.a (mon.0) 85 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-24T17:28:13.881 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:13 smithi032 bash[21457]: cluster 2024-04-24T17:28:13.808746+0000 mon.a (mon.0) 86 : cluster [INF] Manager daemon y is now available 2024-04-24T17:28:13.881 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:13 smithi032 bash[21457]: cluster 2024-04-24T17:28:13.808746+0000 mon.a (mon.0) 86 : cluster [INF] Manager daemon y is now available 2024-04-24T17:28:15.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:14 smithi032 bash[21457]: audit 2024-04-24T17:28:13.833626+0000 mon.a (mon.0) 87 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T17:28:15.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:14 smithi032 bash[21457]: audit 2024-04-24T17:28:13.833626+0000 mon.a (mon.0) 87 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T17:28:15.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:14 smithi032 bash[21457]: audit 2024-04-24T17:28:13.854084+0000 mon.a (mon.0) 88 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T17:28:15.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:14 smithi032 bash[21457]: audit 2024-04-24T17:28:13.854084+0000 mon.a (mon.0) 88 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-24T17:28:15.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:14 smithi032 bash[21457]: audit 2024-04-24T17:28:13.857261+0000 mon.a (mon.0) 89 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T17:28:15.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:14 smithi032 bash[21457]: audit 2024-04-24T17:28:13.857261+0000 mon.a (mon.0) 89 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-24T17:28:15.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:14 smithi032 bash[21457]: cephadm 2024-04-24T17:28:14.048464+0000 mgr.y (mgr.14152) 1 : cephadm [INF] [24/Apr/2024:17:28:14] ENGINE Bus STARTING 2024-04-24T17:28:15.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:14 smithi032 bash[21457]: cephadm 2024-04-24T17:28:14.048464+0000 mgr.y (mgr.14152) 1 : cephadm [INF] [24/Apr/2024:17:28:14] ENGINE Bus STARTING 2024-04-24T17:28:15.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:14 smithi032 bash[21457]: cephadm 2024-04-24T17:28:14.171694+0000 mgr.y (mgr.14152) 2 : cephadm [INF] [24/Apr/2024:17:28:14] ENGINE Serving on https://172.21.15.32:7150 2024-04-24T17:28:15.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:14 smithi032 bash[21457]: cephadm 2024-04-24T17:28:14.171694+0000 mgr.y (mgr.14152) 2 : cephadm [INF] [24/Apr/2024:17:28:14] ENGINE Serving on https://172.21.15.32:7150 2024-04-24T17:28:15.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:14 smithi032 bash[21457]: cephadm 2024-04-24T17:28:14.273055+0000 mgr.y (mgr.14152) 3 : cephadm [INF] [24/Apr/2024:17:28:14] ENGINE Serving on http://172.21.15.32:8765 2024-04-24T17:28:15.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:14 smithi032 bash[21457]: cephadm 2024-04-24T17:28:14.273055+0000 mgr.y (mgr.14152) 3 : cephadm [INF] [24/Apr/2024:17:28:14] ENGINE Serving on http://172.21.15.32:8765 2024-04-24T17:28:15.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:14 smithi032 bash[21457]: cephadm 2024-04-24T17:28:14.273237+0000 mgr.y (mgr.14152) 4 : cephadm [INF] [24/Apr/2024:17:28:14] ENGINE Bus STARTED 2024-04-24T17:28:15.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:14 smithi032 bash[21457]: cephadm 2024-04-24T17:28:14.273237+0000 mgr.y (mgr.14152) 4 : cephadm [INF] [24/Apr/2024:17:28:14] ENGINE Bus STARTED 2024-04-24T17:28:15.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:14 smithi032 bash[21457]: cluster 2024-04-24T17:28:14.787166+0000 mon.a (mon.0) 90 : cluster [DBG] mgrmap e11: y(active, since 1.01337s) 2024-04-24T17:28:15.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:14 smithi032 bash[21457]: cluster 2024-04-24T17:28:14.787166+0000 mon.a (mon.0) 90 : cluster [DBG] mgrmap e11: y(active, since 1.01337s) 2024-04-24T17:28:15.499 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-04-24T17:28:15.499 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-24T17:28:15.499 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-24T17:28:15.499 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-04-24T17:28:15.499 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.968+0000 7f145ceab700 1 Processor -- start 2024-04-24T17:28:15.499 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.968+0000 7f145ceab700 1 -- start start 2024-04-24T17:28:15.500 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.968+0000 7f145ceab700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f145810ab90 con 0x7f1458106760 2024-04-24T17:28:15.500 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.968+0000 7f145659c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f1458106760 0x7f1458106b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:43416/0 (socket says 172.21.15.32:43416) 2024-04-24T17:28:15.500 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.968+0000 7f145659c700 1 -- 172.21.15.32:0/359300391 learned_addr learned my addr 172.21.15.32:0/359300391 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:28:15.500 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.968+0000 7f145559a700 1 -- 172.21.15.32:0/359300391 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1710063027 0 0) 0x7f145810ab90 con 0x7f1458106760 2024-04-24T17:28:15.500 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.968+0000 7f145559a700 1 -- 172.21.15.32:0/359300391 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14400035b0 con 0x7f1458106760 2024-04-24T17:28:15.500 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.972+0000 7f145559a700 1 -- 172.21.15.32:0/359300391 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3989128965 0 0) 0x7f14400035b0 con 0x7f1458106760 2024-04-24T17:28:15.500 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.972+0000 7f145559a700 1 -- 172.21.15.32:0/359300391 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f145810bd70 con 0x7f1458106760 2024-04-24T17:28:15.500 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.972+0000 7f145559a700 1 -- 172.21.15.32:0/359300391 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f1448001900 con 0x7f1458106760 2024-04-24T17:28:15.500 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.972+0000 7f145559a700 1 -- 172.21.15.32:0/359300391 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(25 keys) ==== 1042+0+0 (unknown 2768897130 0 0) 0x7f14480020b0 con 0x7f1458106760 2024-04-24T17:28:15.500 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.972+0000 7f145559a700 1 -- 172.21.15.32:0/359300391 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f1448005ce0 con 0x7f1458106760 2024-04-24T17:28:15.500 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.972+0000 7f145ceab700 1 -- 172.21.15.32:0/359300391 >> v1:172.21.15.32:6789/0 conn(0x7f1458106760 legacy=0x7f1458106b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:28:15.500 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.972+0000 7f145ceab700 1 -- 172.21.15.32:0/359300391 shutdown_connections 2024-04-24T17:28:15.500 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.972+0000 7f145ceab700 1 -- 172.21.15.32:0/359300391 >> 172.21.15.32:0/359300391 conn(0x7f1458101d00 msgr2=0x7f1458104140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:28:15.500 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.972+0000 7f145ceab700 1 -- 172.21.15.32:0/359300391 shutdown_connections 2024-04-24T17:28:15.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.972+0000 7f145ceab700 1 -- 172.21.15.32:0/359300391 wait complete. 2024-04-24T17:28:15.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.972+0000 7f145ceab700 1 librados: starting msgr at 2024-04-24T17:28:15.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.972+0000 7f145ceab700 1 librados: starting objecter 2024-04-24T17:28:15.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.972+0000 7f145ceab700 1 Processor -- start 2024-04-24T17:28:15.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.972+0000 7f145ceab700 1 -- start start 2024-04-24T17:28:15.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.972+0000 7f145ceab700 1 librados: setting wanted keys 2024-04-24T17:28:15.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.972+0000 7f145ceab700 1 librados: calling monclient init 2024-04-24T17:28:15.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.972+0000 7f145ceab700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f145819bbc0 con 0x7f1458106760 2024-04-24T17:28:15.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.972+0000 7f145659c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f1458106760 0x7f145819b4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:43418/0 (socket says 172.21.15.32:43418) 2024-04-24T17:28:15.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.972+0000 7f145659c700 1 -- 172.21.15.32:0/4092907385 learned_addr learned my addr 172.21.15.32:0/4092907385 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:28:15.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.972+0000 7f14477fe700 10 client.?.objecter ms_handle_connect 0x7f1458106760 2024-04-24T17:28:15.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.972+0000 7f14477fe700 10 client.?.objecter resend_mon_ops 2024-04-24T17:28:15.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.972+0000 7f14477fe700 1 -- 172.21.15.32:0/4092907385 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3458118849 0 0) 0x7f145819bbc0 con 0x7f1458106760 2024-04-24T17:28:15.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.972+0000 7f14477fe700 1 -- 172.21.15.32:0/4092907385 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14400056e0 con 0x7f1458106760 2024-04-24T17:28:15.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.972+0000 7f14477fe700 1 -- 172.21.15.32:0/4092907385 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3049673855 0 0) 0x7f145810bd70 con 0x7f1458106760 2024-04-24T17:28:15.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.972+0000 7f14477fe700 1 -- 172.21.15.32:0/4092907385 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f145819bbc0 con 0x7f1458106760 2024-04-24T17:28:15.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.972+0000 7f14477fe700 1 -- 172.21.15.32:0/4092907385 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f1448001400 con 0x7f1458106760 2024-04-24T17:28:15.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.972+0000 7f14477fe700 1 -- 172.21.15.32:0/4092907385 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3309628502 0 0) 0x7f145819bbc0 con 0x7f1458106760 2024-04-24T17:28:15.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.972+0000 7f14477fe700 1 -- 172.21.15.32:0/4092907385 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f145819bd90 con 0x7f1458106760 2024-04-24T17:28:15.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.972+0000 7f145ceab700 1 -- 172.21.15.32:0/4092907385 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f145819c080 con 0x7f1458106760 2024-04-24T17:28:15.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.972+0000 7f145ceab700 10 client.14156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-24T17:28:15.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.972+0000 7f145ceab700 1 -- 172.21.15.32:0/4092907385 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f145819fbd0 con 0x7f1458106760 2024-04-24T17:28:15.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.972+0000 7f145ceab700 1 librados: init done 2024-04-24T17:28:15.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.972+0000 7f14477fe700 1 -- 172.21.15.32:0/4092907385 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(25 keys) ==== 1042+0+0 (unknown 2768897130 0 0) 0x7f1448001fe0 con 0x7f1458106760 2024-04-24T17:28:15.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.972+0000 7f14477fe700 1 -- 172.21.15.32:0/4092907385 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f14480062c0 con 0x7f1458106760 2024-04-24T17:28:15.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.976+0000 7f14477fe700 1 -- 172.21.15.32:0/4092907385 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 9) ==== 46536+0+0 (unknown 3295546111 0 0) 0x7f1448011a30 con 0x7f1458106760 2024-04-24T17:28:15.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.976+0000 7f14477fe700 1 -- 172.21.15.32:0/4092907385 --> v1:172.21.15.32:6800/3410850029 -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f1448001400 con 0x7f144003c460 2024-04-24T17:28:15.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.976+0000 7f14477fe700 1 -- 172.21.15.32:0/4092907385 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 3338368936 0 0) 0x7f1448048230 con 0x7f1458106760 2024-04-24T17:28:15.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.976+0000 7f14477fe700 10 client.14156.objecter ms_dispatch 0x7f1458001550 osd_map(2..2 src has 1..2) 2024-04-24T17:28:15.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.976+0000 7f14477fe700 3 client.14156.objecter handle_osd_map got epochs [2,2] > 0 2024-04-24T17:28:15.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.976+0000 7f14477fe700 3 client.14156.objecter handle_osd_map decoding full epoch 2 2024-04-24T17:28:15.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.976+0000 7f1455d9b700 1 -- 172.21.15.32:0/4092907385 >> v1:172.21.15.32:6800/3410850029 conn(0x7f144003c460 legacy=0x7f144003e920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.32:6800/3410850029 2024-04-24T17:28:15.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:08.976+0000 7f14477fe700 20 client.14156.objecter dump_active .. 0 homeless 2024-04-24T17:28:15.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:09.180+0000 7f1455d9b700 1 -- 172.21.15.32:0/4092907385 >> v1:172.21.15.32:6800/3410850029 conn(0x7f144003c460 legacy=0x7f144003e920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.32:6800/3410850029 2024-04-24T17:28:15.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:09.584+0000 7f1455d9b700 1 -- 172.21.15.32:0/4092907385 >> v1:172.21.15.32:6800/3410850029 conn(0x7f144003c460 legacy=0x7f144003e920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.32:6800/3410850029 2024-04-24T17:28:15.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:10.388+0000 7f1455d9b700 1 -- 172.21.15.32:0/4092907385 >> v1:172.21.15.32:6800/3410850029 conn(0x7f144003c460 legacy=0x7f144003e920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.32:6800/3410850029 2024-04-24T17:28:15.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:11.992+0000 7f1455d9b700 1 -- 172.21.15.32:0/4092907385 >> v1:172.21.15.32:6800/3410850029 conn(0x7f144003c460 legacy=0x7f144003e920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.32:6800/3410850029 2024-04-24T17:28:15.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:13.772+0000 7f14477fe700 1 -- 172.21.15.32:0/4092907385 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mgrmap(e 10) ==== 46338+0+0 (unknown 1258936342 0 0) 0x7f14480138d0 con 0x7f1458106760 2024-04-24T17:28:15.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:13.772+0000 7f14477fe700 1 -- 172.21.15.32:0/4092907385 >> v1:172.21.15.32:6800/3410850029 conn(0x7f144003c460 legacy=0x7f144003e920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T17:28:15.504 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:13.976+0000 7f1457fff700 10 client.14156.objecter tick 2024-04-24T17:28:15.504 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:14.776+0000 7f14477fe700 1 -- 172.21.15.32:0/4092907385 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mgrmap(e 11) ==== 46430+0+0 (unknown 2600558021 0 0) 0x7f1448049230 con 0x7f1458106760 2024-04-24T17:28:15.504 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:14.776+0000 7f14477fe700 1 -- 172.21.15.32:0/4092907385 --> v1:172.21.15.32:6800/1847389373 -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f144802f850 con 0x7f1440040e10 2024-04-24T17:28:15.504 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:14.780+0000 7f14477fe700 10 client.14156.objecter ms_handle_connect 0x7f1440040e10 2024-04-24T17:28:15.504 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:14.780+0000 7f14477fe700 1 -- 172.21.15.32:0/4092907385 <== mgr.14152 v1:172.21.15.32:6800/1847389373 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (unknown 0 0 1361822222) 0x7f144802f850 con 0x7f1440040e10 2024-04-24T17:28:15.504 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:14.796+0000 7f145ceab700 1 -- 172.21.15.32:0/4092907385 --> v1:172.21.15.32:6800/1847389373 -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f1438002630 con 0x7f1440040e10 2024-04-24T17:28:15.504 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:14.796+0000 7f14477fe700 1 -- 172.21.15.32:0/4092907385 <== mgr.14152 v1:172.21.15.32:6800/1847389373 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (unknown 0 0 3086460295) 0x7f1438002630 con 0x7f1440040e10 2024-04-24T17:28:15.504 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:14.796+0000 7f145ceab700 10 librados: watch_flush enter 2024-04-24T17:28:15.504 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:14.796+0000 7f145ceab700 10 librados: watch_flush exit 2024-04-24T17:28:15.504 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:14.796+0000 7f145ceab700 20 client.14156.objecter shutdown clearing up homeless session... 2024-04-24T17:28:15.504 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:14.796+0000 7f145ceab700 10 client.14156.objecter successfully canceled tick 2024-04-24T17:28:15.504 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:14.796+0000 7f145ceab700 1 -- 172.21.15.32:0/4092907385 >> v1:172.21.15.32:6800/1847389373 conn(0x7f1440040e10 legacy=0x7f1440043200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:28:15.504 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:14.796+0000 7f145ceab700 1 -- 172.21.15.32:0/4092907385 >> v1:172.21.15.32:6789/0 conn(0x7f1458106760 legacy=0x7f145819b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:28:15.505 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:14.800+0000 7f145ceab700 1 -- 172.21.15.32:0/4092907385 shutdown_connections 2024-04-24T17:28:15.505 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:14.800+0000 7f145ceab700 1 -- 172.21.15.32:0/4092907385 >> 172.21.15.32:0/4092907385 conn(0x7f1458101d00 msgr2=0x7f1458079fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:28:15.505 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:14.800+0000 7f145ceab700 1 -- 172.21.15.32:0/4092907385 shutdown_connections 2024-04-24T17:28:15.505 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:14.800+0000 7f145ceab700 1 -- 172.21.15.32:0/4092907385 wait complete. 2024-04-24T17:28:15.505 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:14.800+0000 7f145ceab700 1 librados: shutdown 2024-04-24T17:28:15.505 INFO:teuthology.orchestra.run.smithi032.stdout:mgr epoch 9 is available 2024-04-24T17:28:15.505 INFO:teuthology.orchestra.run.smithi032.stdout:Generating a dashboard self-signed certificate... 2024-04-24T17:28:16.151 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:15 smithi032 bash[21457]: audit 2024-04-24T17:28:14.784352+0000 mgr.y (mgr.14152) 5 : audit [DBG] from='client.14156 v1:172.21.15.32:0/4092907385' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-24T17:28:16.152 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:15 smithi032 bash[21457]: audit 2024-04-24T17:28:14.784352+0000 mgr.y (mgr.14152) 5 : audit [DBG] from='client.14156 v1:172.21.15.32:0/4092907385' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-24T17:28:16.152 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:15 smithi032 bash[21457]: audit 2024-04-24T17:28:14.801789+0000 mgr.y (mgr.14152) 6 : audit [DBG] from='client.14156 v1:172.21.15.32:0/4092907385' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-24T17:28:16.152 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:15 smithi032 bash[21457]: audit 2024-04-24T17:28:14.801789+0000 mgr.y (mgr.14152) 6 : audit [DBG] from='client.14156 v1:172.21.15.32:0/4092907385' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-24T17:28:17.145 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:16 smithi032 bash[21457]: cluster 2024-04-24T17:28:15.839042+0000 mon.a (mon.0) 91 : cluster [DBG] mgrmap e12: y(active, since 2s) 2024-04-24T17:28:17.145 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:16 smithi032 bash[21457]: cluster 2024-04-24T17:28:15.839042+0000 mon.a (mon.0) 91 : cluster [DBG] mgrmap e12: y(active, since 2s) 2024-04-24T17:28:17.612 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-24T17:28:17.613 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.355+0000 7fdd433af700 1 Processor -- start 2024-04-24T17:28:17.613 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.355+0000 7fdd433af700 1 -- start start 2024-04-24T17:28:17.613 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.355+0000 7fdd433af700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdd3c10ded0 con 0x7fdd3c109aa0 2024-04-24T17:28:17.613 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.355+0000 7fdd4114b700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fdd3c109aa0 0x7fdd3c109ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58490/0 (socket says 172.21.15.32:58490) 2024-04-24T17:28:17.613 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.355+0000 7fdd4114b700 1 -- 172.21.15.32:0/4120693614 learned_addr learned my addr 172.21.15.32:0/4120693614 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:28:17.613 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.355+0000 7fdd33fff700 1 -- 172.21.15.32:0/4120693614 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3082006309 0 0) 0x7fdd3c10ded0 con 0x7fdd3c109aa0 2024-04-24T17:28:17.613 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.355+0000 7fdd33fff700 1 -- 172.21.15.32:0/4120693614 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd280035b0 con 0x7fdd3c109aa0 2024-04-24T17:28:17.613 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.355+0000 7fdd33fff700 1 -- 172.21.15.32:0/4120693614 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2007026644 0 0) 0x7fdd280035b0 con 0x7fdd3c109aa0 2024-04-24T17:28:17.613 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.355+0000 7fdd33fff700 1 -- 172.21.15.32:0/4120693614 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd3c10f0b0 con 0x7fdd3c109aa0 2024-04-24T17:28:17.614 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.355+0000 7fdd33fff700 1 -- 172.21.15.32:0/4120693614 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fdd38001580 con 0x7fdd3c109aa0 2024-04-24T17:28:17.614 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.355+0000 7fdd33fff700 1 -- 172.21.15.32:0/4120693614 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(25 keys) ==== 1042+0+0 (unknown 2768897130 0 0) 0x7fdd38001c60 con 0x7fdd3c109aa0 2024-04-24T17:28:17.614 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.355+0000 7fdd33fff700 1 -- 172.21.15.32:0/4120693614 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fdd38005780 con 0x7fdd3c109aa0 2024-04-24T17:28:17.614 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.355+0000 7fdd433af700 1 -- 172.21.15.32:0/4120693614 >> v1:172.21.15.32:6789/0 conn(0x7fdd3c109aa0 legacy=0x7fdd3c109ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:28:17.614 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.355+0000 7fdd433af700 1 -- 172.21.15.32:0/4120693614 shutdown_connections 2024-04-24T17:28:17.614 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.355+0000 7fdd433af700 1 -- 172.21.15.32:0/4120693614 >> 172.21.15.32:0/4120693614 conn(0x7fdd3c07ce20 msgr2=0x7fdd3c1074a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:28:17.614 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.355+0000 7fdd433af700 1 -- 172.21.15.32:0/4120693614 shutdown_connections 2024-04-24T17:28:17.614 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.355+0000 7fdd433af700 1 -- 172.21.15.32:0/4120693614 wait complete. 2024-04-24T17:28:17.614 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.355+0000 7fdd433af700 1 librados: starting msgr at 2024-04-24T17:28:17.614 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.355+0000 7fdd433af700 1 librados: starting objecter 2024-04-24T17:28:17.614 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.355+0000 7fdd433af700 1 Processor -- start 2024-04-24T17:28:17.614 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.355+0000 7fdd433af700 1 -- start start 2024-04-24T17:28:17.614 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.355+0000 7fdd433af700 1 librados: setting wanted keys 2024-04-24T17:28:17.615 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.355+0000 7fdd433af700 1 librados: calling monclient init 2024-04-24T17:28:17.615 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.355+0000 7fdd433af700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdd3c19dd50 con 0x7fdd3c109aa0 2024-04-24T17:28:17.615 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.355+0000 7fdd4114b700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fdd3c109aa0 0x7fdd3c19d640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58496/0 (socket says 172.21.15.32:58496) 2024-04-24T17:28:17.615 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.355+0000 7fdd4114b700 1 -- 172.21.15.32:0/1206515030 learned_addr learned my addr 172.21.15.32:0/1206515030 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:28:17.615 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.355+0000 7fdd327fc700 10 client.?.objecter ms_handle_connect 0x7fdd3c109aa0 2024-04-24T17:28:17.615 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.355+0000 7fdd327fc700 10 client.?.objecter resend_mon_ops 2024-04-24T17:28:17.615 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.355+0000 7fdd327fc700 1 -- 172.21.15.32:0/1206515030 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1388170995 0 0) 0x7fdd3c19dd50 con 0x7fdd3c109aa0 2024-04-24T17:28:17.615 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.355+0000 7fdd327fc700 1 -- 172.21.15.32:0/1206515030 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd280035b0 con 0x7fdd3c109aa0 2024-04-24T17:28:17.615 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.359+0000 7fdd327fc700 1 -- 172.21.15.32:0/1206515030 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2117656327 0 0) 0x7fdd280035b0 con 0x7fdd3c109aa0 2024-04-24T17:28:17.615 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.359+0000 7fdd327fc700 1 -- 172.21.15.32:0/1206515030 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdd3c19dd50 con 0x7fdd3c109aa0 2024-04-24T17:28:17.615 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.359+0000 7fdd327fc700 1 -- 172.21.15.32:0/1206515030 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fdd38001080 con 0x7fdd3c109aa0 2024-04-24T17:28:17.615 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.359+0000 7fdd327fc700 1 -- 172.21.15.32:0/1206515030 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4147552081 0 0) 0x7fdd3c19dd50 con 0x7fdd3c109aa0 2024-04-24T17:28:17.615 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.359+0000 7fdd327fc700 1 -- 172.21.15.32:0/1206515030 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd3c19df20 con 0x7fdd3c109aa0 2024-04-24T17:28:17.615 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.359+0000 7fdd433af700 1 -- 172.21.15.32:0/1206515030 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdd3c19e210 con 0x7fdd3c109aa0 2024-04-24T17:28:17.616 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.359+0000 7fdd433af700 10 client.14164.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-24T17:28:17.616 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.359+0000 7fdd433af700 1 -- 172.21.15.32:0/1206515030 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdd3c1a1d60 con 0x7fdd3c109aa0 2024-04-24T17:28:17.616 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.359+0000 7fdd433af700 1 librados: init done 2024-04-24T17:28:17.616 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.359+0000 7fdd327fc700 1 -- 172.21.15.32:0/1206515030 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(25 keys) ==== 1042+0+0 (unknown 2768897130 0 0) 0x7fdd38001c20 con 0x7fdd3c109aa0 2024-04-24T17:28:17.616 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.359+0000 7fdd327fc700 1 -- 172.21.15.32:0/1206515030 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fdd38005ff0 con 0x7fdd3c109aa0 2024-04-24T17:28:17.616 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.359+0000 7fdd327fc700 1 -- 172.21.15.32:0/1206515030 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 12) ==== 46536+0+0 (unknown 1377037603 0 0) 0x7fdd38011820 con 0x7fdd3c109aa0 2024-04-24T17:28:17.616 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.359+0000 7fdd433af700 1 -- 172.21.15.32:0/1206515030 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd20005150 con 0x7fdd3c109aa0 2024-04-24T17:28:17.616 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.359+0000 7fdd327fc700 1 -- 172.21.15.32:0/1206515030 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 1575289653 0 0) 0x7fdd38048220 con 0x7fdd3c109aa0 2024-04-24T17:28:17.616 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.359+0000 7fdd327fc700 10 client.14164.objecter ms_dispatch 0x7fdd3c001550 osd_map(3..3 src has 1..3) 2024-04-24T17:28:17.616 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.359+0000 7fdd327fc700 3 client.14164.objecter handle_osd_map got epochs [3,3] > 0 2024-04-24T17:28:17.616 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.359+0000 7fdd327fc700 3 client.14164.objecter handle_osd_map decoding full epoch 3 2024-04-24T17:28:17.616 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.359+0000 7fdd327fc700 20 client.14164.objecter dump_active .. 0 homeless 2024-04-24T17:28:17.616 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.359+0000 7fdd327fc700 10 client.14164.objecter ms_handle_connect 0x7fdd2803c370 2024-04-24T17:28:17.616 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.363+0000 7fdd327fc700 1 -- 172.21.15.32:0/1206515030 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188728 (unknown 1092875540 0 2656817166) 0x7fdd380177b0 con 0x7fdd3c109aa0 2024-04-24T17:28:17.617 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.767+0000 7fdd433af700 1 -- 172.21.15.32:0/1206515030 --> v1:172.21.15.32:6800/1847389373 -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7fdd20000bc0 con 0x7fdd2803c370 2024-04-24T17:28:17.617 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.967+0000 7fdd327fc700 1 -- 172.21.15.32:0/1206515030 <== mgr.14152 v1:172.21.15.32:6800/1847389373 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (unknown 0 0 3317752739) 0x7fdd20000bc0 con 0x7fdd2803c370 2024-04-24T17:28:17.617 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.971+0000 7fdd433af700 10 librados: watch_flush enter 2024-04-24T17:28:17.617 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.971+0000 7fdd433af700 10 librados: watch_flush exit 2024-04-24T17:28:17.617 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.971+0000 7fdd433af700 20 client.14164.objecter shutdown clearing up homeless session... 2024-04-24T17:28:17.617 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.971+0000 7fdd433af700 10 client.14164.objecter successfully canceled tick 2024-04-24T17:28:17.617 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.971+0000 7fdd433af700 1 -- 172.21.15.32:0/1206515030 >> v1:172.21.15.32:6800/1847389373 conn(0x7fdd2803c370 legacy=0x7fdd2803e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:28:17.617 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.971+0000 7fdd433af700 1 -- 172.21.15.32:0/1206515030 >> v1:172.21.15.32:6789/0 conn(0x7fdd3c109aa0 legacy=0x7fdd3c19d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:28:17.617 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.971+0000 7fdd433af700 1 -- 172.21.15.32:0/1206515030 shutdown_connections 2024-04-24T17:28:17.617 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.971+0000 7fdd433af700 1 -- 172.21.15.32:0/1206515030 >> 172.21.15.32:0/1206515030 conn(0x7fdd3c07ce20 msgr2=0x7fdd3c1074a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:28:17.617 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.971+0000 7fdd433af700 1 -- 172.21.15.32:0/1206515030 shutdown_connections 2024-04-24T17:28:17.617 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.971+0000 7fdd433af700 1 -- 172.21.15.32:0/1206515030 wait complete. 2024-04-24T17:28:17.617 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:16.971+0000 7fdd433af700 1 librados: shutdown 2024-04-24T17:28:17.618 INFO:teuthology.orchestra.run.smithi032.stdout:Creating initial admin user... 2024-04-24T17:28:18.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:17 smithi032 bash[21457]: audit 2024-04-24T17:28:16.774224+0000 mgr.y (mgr.14152) 7 : audit [DBG] from='client.14164 v1:172.21.15.32:0/1206515030' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T17:28:18.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:17 smithi032 bash[21457]: audit 2024-04-24T17:28:16.774224+0000 mgr.y (mgr.14152) 7 : audit [DBG] from='client.14164 v1:172.21.15.32:0/1206515030' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T17:28:18.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:17 smithi032 bash[21457]: audit 2024-04-24T17:28:16.968076+0000 mon.a (mon.0) 92 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:28:18.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:17 smithi032 bash[21457]: audit 2024-04-24T17:28:16.968076+0000 mon.a (mon.0) 92 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:28:18.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:17 smithi032 bash[21457]: audit 2024-04-24T17:28:16.973126+0000 mon.a (mon.0) 93 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:28:18.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:17 smithi032 bash[21457]: audit 2024-04-24T17:28:16.973126+0000 mon.a (mon.0) 93 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:28:19.877 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$h/4f0oh841NU6Za3LuQFL.XzW2ygrCiPZ.byIGB3c1JgLPr//gvmi", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713979699, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-24T17:28:19.877 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.523+0000 7fadb7e17700 1 Processor -- start 2024-04-24T17:28:19.877 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.523+0000 7fadb7e17700 1 -- start start 2024-04-24T17:28:19.877 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.527+0000 7fadb7e17700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fadb007bab0 con 0x7fadb007ceb0 2024-04-24T17:28:19.877 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.527+0000 7fadb5bb3700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fadb007ceb0 0x7fadb007b310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58512/0 (socket says 172.21.15.32:58512) 2024-04-24T17:28:19.877 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.527+0000 7fadb5bb3700 1 -- 172.21.15.32:0/2728389228 learned_addr learned my addr 172.21.15.32:0/2728389228 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:28:19.877 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.527+0000 7fadb4bb1700 1 -- 172.21.15.32:0/2728389228 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 533964731 0 0) 0x7fadb007bab0 con 0x7fadb007ceb0 2024-04-24T17:28:19.878 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.527+0000 7fadb4bb1700 1 -- 172.21.15.32:0/2728389228 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad9c0035b0 con 0x7fadb007ceb0 2024-04-24T17:28:19.878 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.527+0000 7fadb4bb1700 1 -- 172.21.15.32:0/2728389228 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3607019200 0 0) 0x7fad9c0035b0 con 0x7fadb007ceb0 2024-04-24T17:28:19.878 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.527+0000 7fadb4bb1700 1 -- 172.21.15.32:0/2728389228 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadb007bc80 con 0x7fadb007ceb0 2024-04-24T17:28:19.878 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.527+0000 7fadb4bb1700 1 -- 172.21.15.32:0/2728389228 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fadac001580 con 0x7fadb007ceb0 2024-04-24T17:28:19.878 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.527+0000 7fadb4bb1700 1 -- 172.21.15.32:0/2728389228 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(25 keys) ==== 1042+0+0 (unknown 2768897130 0 0) 0x7fadac001c60 con 0x7fadb007ceb0 2024-04-24T17:28:19.878 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.527+0000 7fadb4bb1700 1 -- 172.21.15.32:0/2728389228 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fadac005960 con 0x7fadb007ceb0 2024-04-24T17:28:19.878 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.527+0000 7fadb7e17700 1 -- 172.21.15.32:0/2728389228 >> v1:172.21.15.32:6789/0 conn(0x7fadb007ceb0 legacy=0x7fadb007b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:28:19.878 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.527+0000 7fadb7e17700 1 -- 172.21.15.32:0/2728389228 shutdown_connections 2024-04-24T17:28:19.878 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.527+0000 7fadb7e17700 1 -- 172.21.15.32:0/2728389228 >> 172.21.15.32:0/2728389228 conn(0x7fadb0102e30 msgr2=0x7fadb0105270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:28:19.878 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.527+0000 7fadb7e17700 1 -- 172.21.15.32:0/2728389228 shutdown_connections 2024-04-24T17:28:19.878 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.527+0000 7fadb7e17700 1 -- 172.21.15.32:0/2728389228 wait complete. 2024-04-24T17:28:19.878 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.527+0000 7fadb7e17700 1 librados: starting msgr at 2024-04-24T17:28:19.879 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.527+0000 7fadb7e17700 1 librados: starting objecter 2024-04-24T17:28:19.879 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.527+0000 7fadb7e17700 1 Processor -- start 2024-04-24T17:28:19.879 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.527+0000 7fadb7e17700 1 -- start start 2024-04-24T17:28:19.879 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.527+0000 7fadb7e17700 1 librados: setting wanted keys 2024-04-24T17:28:19.879 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.527+0000 7fadb7e17700 1 librados: calling monclient init 2024-04-24T17:28:19.879 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.527+0000 7fadb7e17700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fadb019ddd0 con 0x7fadb007ceb0 2024-04-24T17:28:19.879 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.527+0000 7fadb5bb3700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fadb007ceb0 0x7fadb019d6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58516/0 (socket says 172.21.15.32:58516) 2024-04-24T17:28:19.879 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.527+0000 7fadb5bb3700 1 -- 172.21.15.32:0/3707075370 learned_addr learned my addr 172.21.15.32:0/3707075370 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:28:19.879 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.527+0000 7fada6ffd700 10 client.?.objecter ms_handle_connect 0x7fadb007ceb0 2024-04-24T17:28:19.879 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.527+0000 7fada6ffd700 10 client.?.objecter resend_mon_ops 2024-04-24T17:28:19.879 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.531+0000 7fada6ffd700 1 -- 172.21.15.32:0/3707075370 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3599824854 0 0) 0x7fadb019ddd0 con 0x7fadb007ceb0 2024-04-24T17:28:19.879 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.531+0000 7fada6ffd700 1 -- 172.21.15.32:0/3707075370 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad9c0035b0 con 0x7fadb007ceb0 2024-04-24T17:28:19.879 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.531+0000 7fada6ffd700 1 -- 172.21.15.32:0/3707075370 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3275660116 0 0) 0x7fad9c0035b0 con 0x7fadb007ceb0 2024-04-24T17:28:19.879 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.531+0000 7fada6ffd700 1 -- 172.21.15.32:0/3707075370 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fadb019ddd0 con 0x7fadb007ceb0 2024-04-24T17:28:19.880 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.531+0000 7fada6ffd700 1 -- 172.21.15.32:0/3707075370 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fadac001080 con 0x7fadb007ceb0 2024-04-24T17:28:19.880 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.531+0000 7fada6ffd700 1 -- 172.21.15.32:0/3707075370 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1099913175 0 0) 0x7fadb019ddd0 con 0x7fadb007ceb0 2024-04-24T17:28:19.880 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.531+0000 7fada6ffd700 1 -- 172.21.15.32:0/3707075370 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadb019dfa0 con 0x7fadb007ceb0 2024-04-24T17:28:19.880 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.531+0000 7fadb7e17700 1 -- 172.21.15.32:0/3707075370 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fadb019e290 con 0x7fadb007ceb0 2024-04-24T17:28:19.880 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.531+0000 7fadb7e17700 10 client.14166.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-24T17:28:19.880 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.531+0000 7fadb7e17700 1 -- 172.21.15.32:0/3707075370 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fadb01a1de0 con 0x7fadb007ceb0 2024-04-24T17:28:19.880 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.531+0000 7fadb7e17700 1 librados: init done 2024-04-24T17:28:19.880 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.531+0000 7fada6ffd700 1 -- 172.21.15.32:0/3707075370 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(25 keys) ==== 1042+0+0 (unknown 2768897130 0 0) 0x7fadac001c20 con 0x7fadb007ceb0 2024-04-24T17:28:19.880 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.531+0000 7fada6ffd700 1 -- 172.21.15.32:0/3707075370 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fadac005fd0 con 0x7fadb007ceb0 2024-04-24T17:28:19.880 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.531+0000 7fada6ffd700 1 -- 172.21.15.32:0/3707075370 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 12) ==== 46536+0+0 (unknown 1377037603 0 0) 0x7fadac006190 con 0x7fadb007ceb0 2024-04-24T17:28:19.880 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.531+0000 7fada6ffd700 1 -- 172.21.15.32:0/3707075370 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 1575289653 0 0) 0x7fadac048e50 con 0x7fadb007ceb0 2024-04-24T17:28:19.880 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.531+0000 7fada6ffd700 10 client.14166.objecter ms_dispatch 0x7fadb0001550 osd_map(3..3 src has 1..3) 2024-04-24T17:28:19.881 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.531+0000 7fada6ffd700 3 client.14166.objecter handle_osd_map got epochs [3,3] > 0 2024-04-24T17:28:19.881 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.531+0000 7fada6ffd700 3 client.14166.objecter handle_osd_map decoding full epoch 3 2024-04-24T17:28:19.881 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.531+0000 7fadb7e17700 1 -- 172.21.15.32:0/3707075370 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad94005150 con 0x7fadb007ceb0 2024-04-24T17:28:19.881 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.531+0000 7fada6ffd700 20 client.14166.objecter dump_active .. 0 homeless 2024-04-24T17:28:19.881 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.531+0000 7fada6ffd700 10 client.14166.objecter ms_handle_connect 0x7fad9c03c460 2024-04-24T17:28:19.881 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.535+0000 7fada6ffd700 1 -- 172.21.15.32:0/3707075370 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188728 (unknown 1092875540 0 2656817166) 0x7fadac017680 con 0x7fadb007ceb0 2024-04-24T17:28:19.881 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:18.959+0000 7fadb7e17700 1 -- 172.21.15.32:0/3707075370 --> v1:172.21.15.32:6800/1847389373 -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7fad94002430 con 0x7fad9c03c460 2024-04-24T17:28:19.881 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:19.223+0000 7fada6ffd700 1 -- 172.21.15.32:0/3707075370 <== mgr.14152 v1:172.21.15.32:6800/1847389373 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (unknown 0 0 2724029260) 0x7fad94002430 con 0x7fad9c03c460 2024-04-24T17:28:19.881 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:19.227+0000 7fadb7e17700 10 librados: watch_flush enter 2024-04-24T17:28:19.881 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:19.227+0000 7fadb7e17700 10 librados: watch_flush exit 2024-04-24T17:28:19.881 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:19.227+0000 7fadb7e17700 20 client.14166.objecter shutdown clearing up homeless session... 2024-04-24T17:28:19.881 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:19.227+0000 7fadb7e17700 10 client.14166.objecter successfully canceled tick 2024-04-24T17:28:19.882 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:19.227+0000 7fadb7e17700 1 -- 172.21.15.32:0/3707075370 >> v1:172.21.15.32:6800/1847389373 conn(0x7fad9c03c460 legacy=0x7fad9c03e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:28:19.882 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:19.227+0000 7fadb7e17700 1 -- 172.21.15.32:0/3707075370 >> v1:172.21.15.32:6789/0 conn(0x7fadb007ceb0 legacy=0x7fadb019d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:28:19.882 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:19.227+0000 7fadb7e17700 1 -- 172.21.15.32:0/3707075370 shutdown_connections 2024-04-24T17:28:19.882 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:19.227+0000 7fadb7e17700 1 -- 172.21.15.32:0/3707075370 >> 172.21.15.32:0/3707075370 conn(0x7fadb0102e30 msgr2=0x7fadb010bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:28:19.882 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:19.227+0000 7fadb7e17700 1 -- 172.21.15.32:0/3707075370 shutdown_connections 2024-04-24T17:28:19.882 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:19.227+0000 7fadb7e17700 1 -- 172.21.15.32:0/3707075370 wait complete. 2024-04-24T17:28:19.882 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:19.227+0000 7fadb7e17700 1 librados: shutdown 2024-04-24T17:28:19.882 INFO:teuthology.orchestra.run.smithi032.stdout:Fetching dashboard port number... 2024-04-24T17:28:20.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:20 smithi032 bash[21457]: audit 2024-04-24T17:28:18.967163+0000 mgr.y (mgr.14152) 8 : audit [DBG] from='client.14166 v1:172.21.15.32:0/3707075370' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T17:28:20.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:20 smithi032 bash[21457]: audit 2024-04-24T17:28:18.967163+0000 mgr.y (mgr.14152) 8 : audit [DBG] from='client.14166 v1:172.21.15.32:0/3707075370' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T17:28:20.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:20 smithi032 bash[21457]: audit 2024-04-24T17:28:19.225303+0000 mon.a (mon.0) 94 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:28:20.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:20 smithi032 bash[21457]: audit 2024-04-24T17:28:19.225303+0000 mon.a (mon.0) 94 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:28:21.229 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:21 smithi032 bash[21457]: cluster 2024-04-24T17:28:20.234988+0000 mon.a (mon.0) 95 : cluster [DBG] mgrmap e13: y(active, since 6s) 2024-04-24T17:28:21.229 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:21 smithi032 bash[21457]: cluster 2024-04-24T17:28:20.234988+0000 mon.a (mon.0) 95 : cluster [DBG] mgrmap e13: y(active, since 6s) 2024-04-24T17:28:21.229 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:21 smithi032 bash[21457]: audit 2024-04-24T17:28:20.603676+0000 mon.a (mon.0) 96 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:28:21.229 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:21 smithi032 bash[21457]: audit 2024-04-24T17:28:20.603676+0000 mon.a (mon.0) 96 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:28:21.229 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:21 smithi032 bash[21457]: audit 2024-04-24T17:28:21.077272+0000 mon.a (mon.0) 97 : audit [DBG] from='client.? v1:172.21.15.32:0/1364691067' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-24T17:28:21.229 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:21 smithi032 bash[21457]: audit 2024-04-24T17:28:21.077272+0000 mon.a (mon.0) 97 : audit [DBG] from='client.? v1:172.21.15.32:0/1364691067' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-24T17:28:21.726 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 8443 2024-04-24T17:28:21.726 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.679+0000 7f046731c700 1 Processor -- start 2024-04-24T17:28:21.726 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.679+0000 7f046731c700 1 -- start start 2024-04-24T17:28:21.726 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.679+0000 7f046731c700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f046007bab0 con 0x7f046007ceb0 2024-04-24T17:28:21.726 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.679+0000 7f04650b8700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f046007ceb0 0x7f046007b310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58522/0 (socket says 172.21.15.32:58522) 2024-04-24T17:28:21.726 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.679+0000 7f04650b8700 1 -- 172.21.15.32:0/3900023388 learned_addr learned my addr 172.21.15.32:0/3900023388 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:28:21.726 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.679+0000 7f045ffff700 1 -- 172.21.15.32:0/3900023388 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 539257233 0 0) 0x7f046007bab0 con 0x7f046007ceb0 2024-04-24T17:28:21.727 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.679+0000 7f045ffff700 1 -- 172.21.15.32:0/3900023388 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04480035b0 con 0x7f046007ceb0 2024-04-24T17:28:21.729 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.679+0000 7f045ffff700 1 -- 172.21.15.32:0/3900023388 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 871084219 0 0) 0x7f04480035b0 con 0x7f046007ceb0 2024-04-24T17:28:21.729 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.679+0000 7f045ffff700 1 -- 172.21.15.32:0/3900023388 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f046007bc80 con 0x7f046007ceb0 2024-04-24T17:28:21.729 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.679+0000 7f045ffff700 1 -- 172.21.15.32:0/3900023388 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f0458001900 con 0x7f046007ceb0 2024-04-24T17:28:21.729 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.679+0000 7f045ffff700 1 -- 172.21.15.32:0/3900023388 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(25 keys) ==== 1042+0+0 (unknown 2768897130 0 0) 0x7f04580020b0 con 0x7f046007ceb0 2024-04-24T17:28:21.730 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.679+0000 7f045ffff700 1 -- 172.21.15.32:0/3900023388 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f0458005ce0 con 0x7f046007ceb0 2024-04-24T17:28:21.730 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.683+0000 7f046731c700 1 -- 172.21.15.32:0/3900023388 >> v1:172.21.15.32:6789/0 conn(0x7f046007ceb0 legacy=0x7f046007b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:28:21.730 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.683+0000 7f046731c700 1 -- 172.21.15.32:0/3900023388 shutdown_connections 2024-04-24T17:28:21.730 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.683+0000 7f046731c700 1 -- 172.21.15.32:0/3900023388 >> 172.21.15.32:0/3900023388 conn(0x7f0460102dc0 msgr2=0x7f0460105240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:28:21.730 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.683+0000 7f046731c700 1 -- 172.21.15.32:0/3900023388 shutdown_connections 2024-04-24T17:28:21.730 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.683+0000 7f046731c700 1 -- 172.21.15.32:0/3900023388 wait complete. 2024-04-24T17:28:21.730 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.683+0000 7f046731c700 1 librados: starting msgr at 2024-04-24T17:28:21.730 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.683+0000 7f046731c700 1 librados: starting objecter 2024-04-24T17:28:21.730 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.683+0000 7f046731c700 1 Processor -- start 2024-04-24T17:28:21.730 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.683+0000 7f046731c700 1 -- start start 2024-04-24T17:28:21.730 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.683+0000 7f046731c700 1 librados: setting wanted keys 2024-04-24T17:28:21.730 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.683+0000 7f046731c700 1 librados: calling monclient init 2024-04-24T17:28:21.730 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.683+0000 7f046731c700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f046019dde0 con 0x7f046007ceb0 2024-04-24T17:28:21.731 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.683+0000 7f04650b8700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f046007ceb0 0x7f046019d6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58532/0 (socket says 172.21.15.32:58532) 2024-04-24T17:28:21.731 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.683+0000 7f04650b8700 1 -- 172.21.15.32:0/1364691067 learned_addr learned my addr 172.21.15.32:0/1364691067 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:28:21.731 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.683+0000 7f045e7fc700 10 client.?.objecter ms_handle_connect 0x7f046007ceb0 2024-04-24T17:28:21.731 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.683+0000 7f045e7fc700 10 client.?.objecter resend_mon_ops 2024-04-24T17:28:21.731 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.683+0000 7f045e7fc700 1 -- 172.21.15.32:0/1364691067 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 419035997 0 0) 0x7f046019dde0 con 0x7f046007ceb0 2024-04-24T17:28:21.731 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.683+0000 7f045e7fc700 1 -- 172.21.15.32:0/1364691067 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04480056e0 con 0x7f046007ceb0 2024-04-24T17:28:21.731 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.683+0000 7f045e7fc700 1 -- 172.21.15.32:0/1364691067 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2805991332 0 0) 0x7f046007bc80 con 0x7f046007ceb0 2024-04-24T17:28:21.734 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.683+0000 7f045e7fc700 1 -- 172.21.15.32:0/1364691067 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f046019dde0 con 0x7f046007ceb0 2024-04-24T17:28:21.734 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.683+0000 7f045e7fc700 1 -- 172.21.15.32:0/1364691067 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f0458001400 con 0x7f046007ceb0 2024-04-24T17:28:21.735 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.683+0000 7f045e7fc700 1 -- 172.21.15.32:0/1364691067 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2706105717 0 0) 0x7f046019dde0 con 0x7f046007ceb0 2024-04-24T17:28:21.735 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.683+0000 7f045e7fc700 1 -- 172.21.15.32:0/1364691067 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f046019dfb0 con 0x7f046007ceb0 2024-04-24T17:28:21.735 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.683+0000 7f046731c700 1 -- 172.21.15.32:0/1364691067 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f046019e2a0 con 0x7f046007ceb0 2024-04-24T17:28:21.735 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.683+0000 7f046731c700 10 client.14168.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-24T17:28:21.735 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.683+0000 7f046731c700 1 -- 172.21.15.32:0/1364691067 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f04601a1df0 con 0x7f046007ceb0 2024-04-24T17:28:21.735 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.683+0000 7f046731c700 1 librados: init done 2024-04-24T17:28:21.735 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.683+0000 7f045e7fc700 1 -- 172.21.15.32:0/1364691067 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(25 keys) ==== 1042+0+0 (unknown 2768897130 0 0) 0x7f04580020b0 con 0x7f046007ceb0 2024-04-24T17:28:21.735 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.683+0000 7f045e7fc700 1 -- 172.21.15.32:0/1364691067 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f0458006410 con 0x7f046007ceb0 2024-04-24T17:28:21.735 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.683+0000 7f045e7fc700 1 -- 172.21.15.32:0/1364691067 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 13) ==== 46579+0+0 (unknown 583022566 0 0) 0x7f0458011cb0 con 0x7f046007ceb0 2024-04-24T17:28:21.735 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.683+0000 7f046731c700 1 -- 172.21.15.32:0/1364691067 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f044c005150 con 0x7f046007ceb0 2024-04-24T17:28:21.735 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.683+0000 7f045e7fc700 1 -- 172.21.15.32:0/1364691067 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 1575289653 0 0) 0x7f0458048600 con 0x7f046007ceb0 2024-04-24T17:28:21.735 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.683+0000 7f045e7fc700 10 client.14168.objecter ms_dispatch 0x7f0460001550 osd_map(3..3 src has 1..3) 2024-04-24T17:28:21.735 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.683+0000 7f045e7fc700 3 client.14168.objecter handle_osd_map got epochs [3,3] > 0 2024-04-24T17:28:21.735 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.683+0000 7f045e7fc700 3 client.14168.objecter handle_osd_map decoding full epoch 3 2024-04-24T17:28:21.736 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.683+0000 7f045e7fc700 20 client.14168.objecter dump_active .. 0 homeless 2024-04-24T17:28:21.736 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.687+0000 7f045e7fc700 10 client.14168.objecter ms_handle_connect 0x7f044803c4b0 2024-04-24T17:28:21.736 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:20.691+0000 7f045e7fc700 1 -- 172.21.15.32:0/1364691067 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188728 (unknown 1092875540 0 2656817166) 0x7f04480056e0 con 0x7f046007ceb0 2024-04-24T17:28:21.736 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:21.071+0000 7f046731c700 1 -- 172.21.15.32:0/1364691067 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f044c005440 con 0x7f046007ceb0 2024-04-24T17:28:21.736 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:21.071+0000 7f045e7fc700 1 -- 172.21.15.32:0/1364691067 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (unknown 3713421687 0 83753974) 0x7f0458017c30 con 0x7f046007ceb0 2024-04-24T17:28:21.736 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:21.075+0000 7f046731c700 10 librados: watch_flush enter 2024-04-24T17:28:21.736 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:21.075+0000 7f046731c700 10 librados: watch_flush exit 2024-04-24T17:28:21.736 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:21.075+0000 7f046731c700 20 client.14168.objecter shutdown clearing up homeless session... 2024-04-24T17:28:21.736 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:21.075+0000 7f046731c700 10 client.14168.objecter successfully canceled tick 2024-04-24T17:28:21.736 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:21.075+0000 7f046731c700 1 -- 172.21.15.32:0/1364691067 >> v1:172.21.15.32:6800/1847389373 conn(0x7f044803c4b0 legacy=0x7f044803e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:28:21.736 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:21.075+0000 7f046731c700 1 -- 172.21.15.32:0/1364691067 >> v1:172.21.15.32:6789/0 conn(0x7f046007ceb0 legacy=0x7f046019d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:28:21.736 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:21.075+0000 7f046731c700 1 -- 172.21.15.32:0/1364691067 shutdown_connections 2024-04-24T17:28:21.736 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:21.075+0000 7f046731c700 1 -- 172.21.15.32:0/1364691067 >> 172.21.15.32:0/1364691067 conn(0x7f0460102dc0 msgr2=0x7f046010bcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:28:21.736 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:21.075+0000 7f046731c700 1 -- 172.21.15.32:0/1364691067 shutdown_connections 2024-04-24T17:28:21.736 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:21.075+0000 7f046731c700 1 -- 172.21.15.32:0/1364691067 wait complete. 2024-04-24T17:28:21.737 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:21.075+0000 7f046731c700 1 librados: shutdown 2024-04-24T17:28:21.737 INFO:teuthology.orchestra.run.smithi032.stdout:firewalld does not appear to be present 2024-04-24T17:28:21.737 INFO:teuthology.orchestra.run.smithi032.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-24T17:28:21.738 INFO:teuthology.orchestra.run.smithi032.stdout:Ceph Dashboard is now available at: 2024-04-24T17:28:21.738 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T17:28:21.738 INFO:teuthology.orchestra.run.smithi032.stdout: URL: https://smithi032.front.sepia.ceph.com:8443/ 2024-04-24T17:28:21.738 INFO:teuthology.orchestra.run.smithi032.stdout: User: admin 2024-04-24T17:28:21.738 INFO:teuthology.orchestra.run.smithi032.stdout: Password: l2vw2yrpxz 2024-04-24T17:28:21.738 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T17:28:21.738 INFO:teuthology.orchestra.run.smithi032.stdout:Saving cluster configuration to /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/config directory 2024-04-24T17:28:21.739 INFO:teuthology.orchestra.run.smithi032.stdout:Enabling autotune for osd_memory_target 2024-04-24T17:28:22.752 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:22 smithi032 bash[21457]: audit 2024-04-24T17:28:21.591682+0000 mon.a (mon.0) 98 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:28:22.752 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:22 smithi032 bash[21457]: audit 2024-04-24T17:28:21.591682+0000 mon.a (mon.0) 98 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:28:23.734 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.643+0000 7fa954c3c700 1 Processor -- start 2024-04-24T17:28:23.734 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.643+0000 7fa954c3c700 1 -- start start 2024-04-24T17:28:23.734 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.643+0000 7fa954c3c700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa95010bc90 con 0x7fa950107860 2024-04-24T17:28:23.734 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.643+0000 7fa94e59c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa950107860 0x7fa950107c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58548/0 (socket says 172.21.15.32:58548) 2024-04-24T17:28:23.734 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.643+0000 7fa94e59c700 1 -- 172.21.15.32:0/2605196584 learned_addr learned my addr 172.21.15.32:0/2605196584 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:28:23.734 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.643+0000 7fa94d59a700 1 -- 172.21.15.32:0/2605196584 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4204067115 0 0) 0x7fa95010bc90 con 0x7fa950107860 2024-04-24T17:28:23.735 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.643+0000 7fa94d59a700 1 -- 172.21.15.32:0/2605196584 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9380035b0 con 0x7fa950107860 2024-04-24T17:28:23.735 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.643+0000 7fa94d59a700 1 -- 172.21.15.32:0/2605196584 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2174813628 0 0) 0x7fa9380035b0 con 0x7fa950107860 2024-04-24T17:28:23.735 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.643+0000 7fa94d59a700 1 -- 172.21.15.32:0/2605196584 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa95010ce70 con 0x7fa950107860 2024-04-24T17:28:23.738 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.643+0000 7fa94d59a700 1 -- 172.21.15.32:0/2605196584 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fa948001900 con 0x7fa950107860 2024-04-24T17:28:23.738 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.643+0000 7fa94d59a700 1 -- 172.21.15.32:0/2605196584 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(25 keys) ==== 1042+0+0 (unknown 2768897130 0 0) 0x7fa948001fe0 con 0x7fa950107860 2024-04-24T17:28:23.739 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.643+0000 7fa94d59a700 1 -- 172.21.15.32:0/2605196584 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fa948005b00 con 0x7fa950107860 2024-04-24T17:28:23.739 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.643+0000 7fa954c3c700 1 -- 172.21.15.32:0/2605196584 >> v1:172.21.15.32:6789/0 conn(0x7fa950107860 legacy=0x7fa950107c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:28:23.739 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.643+0000 7fa954c3c700 1 -- 172.21.15.32:0/2605196584 shutdown_connections 2024-04-24T17:28:23.739 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.643+0000 7fa954c3c700 1 -- 172.21.15.32:0/2605196584 >> 172.21.15.32:0/2605196584 conn(0x7fa950102e00 msgr2=0x7fa950105240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:28:23.739 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.643+0000 7fa954c3c700 1 -- 172.21.15.32:0/2605196584 shutdown_connections 2024-04-24T17:28:23.739 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.643+0000 7fa954c3c700 1 -- 172.21.15.32:0/2605196584 wait complete. 2024-04-24T17:28:23.739 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.643+0000 7fa954c3c700 1 librados: starting msgr at 2024-04-24T17:28:23.739 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.643+0000 7fa954c3c700 1 librados: starting objecter 2024-04-24T17:28:23.739 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.643+0000 7fa954c3c700 1 Processor -- start 2024-04-24T17:28:23.739 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.643+0000 7fa954c3c700 1 -- start start 2024-04-24T17:28:23.739 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.643+0000 7fa954c3c700 1 librados: setting wanted keys 2024-04-24T17:28:23.740 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.643+0000 7fa954c3c700 1 librados: calling monclient init 2024-04-24T17:28:23.740 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.643+0000 7fa954c3c700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa9501a14c0 con 0x7fa950107860 2024-04-24T17:28:23.740 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.647+0000 7fa94e59c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa950107860 0x7fa95007cb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58564/0 (socket says 172.21.15.32:58564) 2024-04-24T17:28:23.740 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.647+0000 7fa94e59c700 1 -- 172.21.15.32:0/4096895827 learned_addr learned my addr 172.21.15.32:0/4096895827 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:28:23.740 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.647+0000 7fa93f7fe700 10 client.?.objecter ms_handle_connect 0x7fa950107860 2024-04-24T17:28:23.740 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.647+0000 7fa93f7fe700 10 client.?.objecter resend_mon_ops 2024-04-24T17:28:23.740 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.647+0000 7fa93f7fe700 1 -- 172.21.15.32:0/4096895827 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 592335456 0 0) 0x7fa9501a14c0 con 0x7fa950107860 2024-04-24T17:28:23.740 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.647+0000 7fa93f7fe700 1 -- 172.21.15.32:0/4096895827 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9380035b0 con 0x7fa950107860 2024-04-24T17:28:23.740 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.647+0000 7fa93f7fe700 1 -- 172.21.15.32:0/4096895827 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2928903407 0 0) 0x7fa9380035b0 con 0x7fa950107860 2024-04-24T17:28:23.740 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.647+0000 7fa93f7fe700 1 -- 172.21.15.32:0/4096895827 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa9501a14c0 con 0x7fa950107860 2024-04-24T17:28:23.740 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.647+0000 7fa93f7fe700 1 -- 172.21.15.32:0/4096895827 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fa948001400 con 0x7fa950107860 2024-04-24T17:28:23.740 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.647+0000 7fa93f7fe700 1 -- 172.21.15.32:0/4096895827 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2247519446 0 0) 0x7fa9501a14c0 con 0x7fa950107860 2024-04-24T17:28:23.740 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.647+0000 7fa93f7fe700 1 -- 172.21.15.32:0/4096895827 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9501a26a0 con 0x7fa950107860 2024-04-24T17:28:23.741 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.647+0000 7fa954c3c700 1 -- 172.21.15.32:0/4096895827 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa9501a16f0 con 0x7fa950107860 2024-04-24T17:28:23.741 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.647+0000 7fa954c3c700 10 client.14170.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-24T17:28:23.741 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.647+0000 7fa954c3c700 1 -- 172.21.15.32:0/4096895827 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa9501a1c30 con 0x7fa950107860 2024-04-24T17:28:23.746 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.647+0000 7fa954c3c700 1 librados: init done 2024-04-24T17:28:23.746 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.647+0000 7fa93f7fe700 1 -- 172.21.15.32:0/4096895827 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(25 keys) ==== 1042+0+0 (unknown 2768897130 0 0) 0x7fa948005a90 con 0x7fa950107860 2024-04-24T17:28:23.746 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.647+0000 7fa93f7fe700 1 -- 172.21.15.32:0/4096895827 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fa948006410 con 0x7fa950107860 2024-04-24T17:28:23.746 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.647+0000 7fa93f7fe700 1 -- 172.21.15.32:0/4096895827 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 13) ==== 46579+0+0 (unknown 583022566 0 0) 0x7fa948011bf0 con 0x7fa950107860 2024-04-24T17:28:23.746 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.647+0000 7fa954c3c700 1 -- 172.21.15.32:0/4096895827 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa930005150 con 0x7fa950107860 2024-04-24T17:28:23.747 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.647+0000 7fa93f7fe700 1 -- 172.21.15.32:0/4096895827 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 1575289653 0 0) 0x7fa948049460 con 0x7fa950107860 2024-04-24T17:28:23.747 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.647+0000 7fa93f7fe700 10 client.14170.objecter ms_dispatch 0x7fa950001550 osd_map(3..3 src has 1..3) 2024-04-24T17:28:23.747 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.647+0000 7fa93f7fe700 3 client.14170.objecter handle_osd_map got epochs [3,3] > 0 2024-04-24T17:28:23.747 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.647+0000 7fa93f7fe700 3 client.14170.objecter handle_osd_map decoding full epoch 3 2024-04-24T17:28:23.747 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.647+0000 7fa93f7fe700 20 client.14170.objecter dump_active .. 0 homeless 2024-04-24T17:28:23.747 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.647+0000 7fa93f7fe700 10 client.14170.objecter ms_handle_connect 0x7fa93803c420 2024-04-24T17:28:23.747 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:22.651+0000 7fa93f7fe700 1 -- 172.21.15.32:0/4096895827 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188728 (unknown 1092875540 0 2656817166) 0x7fa948017c30 con 0x7fa950107860 2024-04-24T17:28:23.747 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:23.035+0000 7fa954c3c700 1 -- 172.21.15.32:0/4096895827 --> v1:172.21.15.32:6789/0 -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7fa930005440 con 0x7fa950107860 2024-04-24T17:28:23.747 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:23.035+0000 7fa93f7fe700 1 -- 172.21.15.32:0/4096895827 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (unknown 689008156 0 0) 0x7fa948017530 con 0x7fa950107860 2024-04-24T17:28:23.747 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:23.039+0000 7fa954c3c700 10 librados: watch_flush enter 2024-04-24T17:28:23.747 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:23.039+0000 7fa954c3c700 10 librados: watch_flush exit 2024-04-24T17:28:23.747 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:23.039+0000 7fa954c3c700 20 client.14170.objecter shutdown clearing up homeless session... 2024-04-24T17:28:23.747 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:23.039+0000 7fa954c3c700 10 client.14170.objecter successfully canceled tick 2024-04-24T17:28:23.747 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:23.039+0000 7fa954c3c700 1 -- 172.21.15.32:0/4096895827 >> v1:172.21.15.32:6800/1847389373 conn(0x7fa93803c420 legacy=0x7fa93803e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:28:23.748 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:23.039+0000 7fa954c3c700 1 -- 172.21.15.32:0/4096895827 >> v1:172.21.15.32:6789/0 conn(0x7fa950107860 legacy=0x7fa95007cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:28:23.748 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:23.039+0000 7fa954c3c700 1 -- 172.21.15.32:0/4096895827 shutdown_connections 2024-04-24T17:28:23.748 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:23.039+0000 7fa954c3c700 1 -- 172.21.15.32:0/4096895827 >> 172.21.15.32:0/4096895827 conn(0x7fa950102e00 msgr2=0x7fa950105240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:28:23.748 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:23.039+0000 7fa954c3c700 1 -- 172.21.15.32:0/4096895827 shutdown_connections 2024-04-24T17:28:23.748 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:23.039+0000 7fa954c3c700 1 -- 172.21.15.32:0/4096895827 wait complete. 2024-04-24T17:28:23.748 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:23.039+0000 7fa954c3c700 1 librados: shutdown 2024-04-24T17:28:25.828 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.575+0000 7f28c1235700 1 Processor -- start 2024-04-24T17:28:25.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.575+0000 7f28c1235700 1 -- start start 2024-04-24T17:28:25.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.575+0000 7f28c1235700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f28bc07ba70 con 0x7f28bc07ce70 2024-04-24T17:28:25.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.575+0000 7f28bad9d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f28bc07ce70 0x7f28bc07b2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58566/0 (socket says 172.21.15.32:58566) 2024-04-24T17:28:25.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.575+0000 7f28bad9d700 1 -- 172.21.15.32:0/4262684166 learned_addr learned my addr 172.21.15.32:0/4262684166 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:28:25.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.575+0000 7f28b9d9b700 1 -- 172.21.15.32:0/4262684166 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 966299600 0 0) 0x7f28bc07ba70 con 0x7f28bc07ce70 2024-04-24T17:28:25.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.575+0000 7f28b9d9b700 1 -- 172.21.15.32:0/4262684166 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f28a40035b0 con 0x7f28bc07ce70 2024-04-24T17:28:25.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.575+0000 7f28b9d9b700 1 -- 172.21.15.32:0/4262684166 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 267564853 0 0) 0x7f28a40035b0 con 0x7f28bc07ce70 2024-04-24T17:28:25.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.575+0000 7f28b9d9b700 1 -- 172.21.15.32:0/4262684166 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28bc07bc40 con 0x7f28bc07ce70 2024-04-24T17:28:25.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.575+0000 7f28b9d9b700 1 -- 172.21.15.32:0/4262684166 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f28ac001580 con 0x7f28bc07ce70 2024-04-24T17:28:25.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.575+0000 7f28b9d9b700 1 -- 172.21.15.32:0/4262684166 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(25 keys) ==== 1042+0+0 (unknown 2768897130 0 0) 0x7f28ac001d30 con 0x7f28bc07ce70 2024-04-24T17:28:25.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.575+0000 7f28b9d9b700 1 -- 172.21.15.32:0/4262684166 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f28ac005960 con 0x7f28bc07ce70 2024-04-24T17:28:25.829 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.579+0000 7f28c1235700 1 -- 172.21.15.32:0/4262684166 >> v1:172.21.15.32:6789/0 conn(0x7f28bc07ce70 legacy=0x7f28bc07b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:28:25.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.579+0000 7f28c1235700 1 -- 172.21.15.32:0/4262684166 shutdown_connections 2024-04-24T17:28:25.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.579+0000 7f28c1235700 1 -- 172.21.15.32:0/4262684166 >> 172.21.15.32:0/4262684166 conn(0x7f28bc102e30 msgr2=0x7f28bc105270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:28:25.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.579+0000 7f28c1235700 1 -- 172.21.15.32:0/4262684166 shutdown_connections 2024-04-24T17:28:25.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.579+0000 7f28c1235700 1 -- 172.21.15.32:0/4262684166 wait complete. 2024-04-24T17:28:25.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.579+0000 7f28c1235700 1 librados: starting msgr at 2024-04-24T17:28:25.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.579+0000 7f28c1235700 1 librados: starting objecter 2024-04-24T17:28:25.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.579+0000 7f28c1235700 1 Processor -- start 2024-04-24T17:28:25.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.579+0000 7f28c1235700 1 -- start start 2024-04-24T17:28:25.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.579+0000 7f28c1235700 1 librados: setting wanted keys 2024-04-24T17:28:25.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.579+0000 7f28c1235700 1 librados: calling monclient init 2024-04-24T17:28:25.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.579+0000 7f28c1235700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f28bc19ddb0 con 0x7f28bc07ce70 2024-04-24T17:28:25.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.579+0000 7f28bad9d700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f28bc07ce70 0x7f28bc19d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:58572/0 (socket says 172.21.15.32:58572) 2024-04-24T17:28:25.830 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.579+0000 7f28bad9d700 1 -- 172.21.15.32:0/713000372 learned_addr learned my addr 172.21.15.32:0/713000372 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:28:25.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.579+0000 7f28a3fff700 10 client.?.objecter ms_handle_connect 0x7f28bc07ce70 2024-04-24T17:28:25.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.579+0000 7f28a3fff700 10 client.?.objecter resend_mon_ops 2024-04-24T17:28:25.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.579+0000 7f28a3fff700 1 -- 172.21.15.32:0/713000372 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2956788625 0 0) 0x7f28bc19ddb0 con 0x7f28bc07ce70 2024-04-24T17:28:25.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.579+0000 7f28a3fff700 1 -- 172.21.15.32:0/713000372 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f28a40056e0 con 0x7f28bc07ce70 2024-04-24T17:28:25.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.579+0000 7f28a3fff700 1 -- 172.21.15.32:0/713000372 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3332805694 0 0) 0x7f28bc07bc40 con 0x7f28bc07ce70 2024-04-24T17:28:25.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.579+0000 7f28a3fff700 1 -- 172.21.15.32:0/713000372 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f28bc19ddb0 con 0x7f28bc07ce70 2024-04-24T17:28:25.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.579+0000 7f28a3fff700 1 -- 172.21.15.32:0/713000372 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f28ac001080 con 0x7f28bc07ce70 2024-04-24T17:28:25.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.579+0000 7f28a3fff700 1 -- 172.21.15.32:0/713000372 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1089171810 0 0) 0x7f28bc19ddb0 con 0x7f28bc07ce70 2024-04-24T17:28:25.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.579+0000 7f28a3fff700 1 -- 172.21.15.32:0/713000372 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28bc19df80 con 0x7f28bc07ce70 2024-04-24T17:28:25.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.579+0000 7f28c1235700 1 -- 172.21.15.32:0/713000372 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f28bc19e270 con 0x7f28bc07ce70 2024-04-24T17:28:25.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.579+0000 7f28c1235700 10 client.14172.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-24T17:28:25.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.579+0000 7f28c1235700 1 -- 172.21.15.32:0/713000372 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f28bc1a1dc0 con 0x7f28bc07ce70 2024-04-24T17:28:25.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.579+0000 7f28c1235700 1 librados: init done 2024-04-24T17:28:25.831 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.579+0000 7f28a3fff700 1 -- 172.21.15.32:0/713000372 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(25 keys) ==== 1042+0+0 (unknown 2768897130 0 0) 0x7f28ac001c60 con 0x7f28bc07ce70 2024-04-24T17:28:25.832 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.579+0000 7f28a3fff700 1 -- 172.21.15.32:0/713000372 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f28ac005f90 con 0x7f28bc07ce70 2024-04-24T17:28:25.832 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.579+0000 7f28c1235700 1 -- 172.21.15.32:0/713000372 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28bc10ef00 con 0x7f28bc07ce70 2024-04-24T17:28:25.832 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.579+0000 7f28a3fff700 1 -- 172.21.15.32:0/713000372 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 13) ==== 46579+0+0 (unknown 583022566 0 0) 0x7f28ac011830 con 0x7f28bc07ce70 2024-04-24T17:28:25.832 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.579+0000 7f28a3fff700 1 -- 172.21.15.32:0/713000372 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 1575289653 0 0) 0x7f28ac048fc0 con 0x7f28bc07ce70 2024-04-24T17:28:25.832 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.579+0000 7f28a3fff700 10 client.14172.objecter ms_dispatch 0x7f28bc001550 osd_map(3..3 src has 1..3) 2024-04-24T17:28:25.832 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.579+0000 7f28a3fff700 3 client.14172.objecter handle_osd_map got epochs [3,3] > 0 2024-04-24T17:28:25.832 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.579+0000 7f28a3fff700 3 client.14172.objecter handle_osd_map decoding full epoch 3 2024-04-24T17:28:25.832 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.583+0000 7f28a3fff700 20 client.14172.objecter dump_active .. 0 homeless 2024-04-24T17:28:25.832 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.583+0000 7f28a3fff700 10 client.14172.objecter ms_handle_connect 0x7f28a403c5d0 2024-04-24T17:28:25.832 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:24.587+0000 7f28a3fff700 1 -- 172.21.15.32:0/713000372 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188728 (unknown 1092875540 0 2656817166) 0x7f28a40056e0 con 0x7f28bc07ce70 2024-04-24T17:28:25.832 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:25.199+0000 7f28c1235700 1 -- 172.21.15.32:0/713000372 --> v1:172.21.15.32:6789/0 -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f28bc10f180 con 0x7f28bc07ce70 2024-04-24T17:28:25.832 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:25.203+0000 7f28a3fff700 1 -- 172.21.15.32:0/713000372 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v26)=0 set mgr/dashboard/cluster/status v26) ==== 153+0+0 (unknown 1951093493 0 0) 0x7f28ac017870 con 0x7f28bc07ce70 2024-04-24T17:28:25.832 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:25.207+0000 7f28c1235700 10 librados: watch_flush enter 2024-04-24T17:28:25.832 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:25.207+0000 7f28c1235700 10 librados: watch_flush exit 2024-04-24T17:28:25.833 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:25.207+0000 7f28c1235700 20 client.14172.objecter shutdown clearing up homeless session... 2024-04-24T17:28:25.833 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:25.207+0000 7f28c1235700 10 client.14172.objecter successfully canceled tick 2024-04-24T17:28:25.833 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:25.207+0000 7f28c1235700 1 -- 172.21.15.32:0/713000372 >> v1:172.21.15.32:6800/1847389373 conn(0x7f28a403c5d0 legacy=0x7f28a403ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:28:25.833 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:25.207+0000 7f28c1235700 1 -- 172.21.15.32:0/713000372 >> v1:172.21.15.32:6789/0 conn(0x7f28bc07ce70 legacy=0x7f28bc19d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:28:25.833 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:25.207+0000 7f28c1235700 1 -- 172.21.15.32:0/713000372 shutdown_connections 2024-04-24T17:28:25.833 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:25.207+0000 7f28c1235700 1 -- 172.21.15.32:0/713000372 >> 172.21.15.32:0/713000372 conn(0x7f28bc102e30 msgr2=0x7f28bc10bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:28:25.833 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:25.207+0000 7f28c1235700 1 -- 172.21.15.32:0/713000372 shutdown_connections 2024-04-24T17:28:25.833 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:25.207+0000 7f28c1235700 1 -- 172.21.15.32:0/713000372 wait complete. 2024-04-24T17:28:25.833 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-24T17:28:25.207+0000 7f28c1235700 1 librados: shutdown 2024-04-24T17:28:25.833 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-24T17:28:25.833 INFO:teuthology.orchestra.run.smithi032.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-24T17:28:25.833 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T17:28:25.833 INFO:teuthology.orchestra.run.smithi032.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid a7981b84-025f-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-24T17:28:25.833 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T17:28:25.834 INFO:teuthology.orchestra.run.smithi032.stdout:Or, if you are only running a single cluster on this host: 2024-04-24T17:28:25.834 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T17:28:25.834 INFO:teuthology.orchestra.run.smithi032.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-24T17:28:25.834 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T17:28:25.834 INFO:teuthology.orchestra.run.smithi032.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-24T17:28:25.834 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T17:28:25.834 INFO:teuthology.orchestra.run.smithi032.stdout: ceph telemetry on 2024-04-24T17:28:25.834 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T17:28:25.834 INFO:teuthology.orchestra.run.smithi032.stdout:For more information see: 2024-04-24T17:28:25.834 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T17:28:25.834 INFO:teuthology.orchestra.run.smithi032.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-24T17:28:25.834 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T17:28:25.834 INFO:teuthology.orchestra.run.smithi032.stdout:Bootstrap complete. 2024-04-24T17:28:25.890 INFO:tasks.cephadm:Fetching config... 2024-04-24T17:28:25.890 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-24T17:28:25.890 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-24T17:28:25.897 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-24T17:28:25.897 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-24T17:28:25.897 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-24T17:28:25.946 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-24T17:28:25.946 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-24T17:28:25.946 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-24T17:28:26.001 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-24T17:28:26.002 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-24T17:28:26.002 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-24T17:28:26.050 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-24T17:28:26.050 DEBUG:teuthology.orchestra.run.smithi032:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDWf4PvC0e2ZX4vQ31byvDugW7gt2B+a2H2ImNvG/O3jM0WV7pTG/P+8oINdEVDHs1JOFjdNZ+ao8GjgObqNonUX0RtfYTqVoekHG2JA2AYl8Qaj8rm8xmVuyrjizHkxvWiDw1URlh2EvUTRD7SpedPsRAcS2bYDzn99wlptBAN8G/LfNnvqa0eabAb7IA7eqHmyp5mRAI0YfB/jOTpEsI5RWpdSJY43Xvj7oIjXcuBzyCey69M57Zm4XP1LW7EE+DZ0OZsNJGEZGxRvKgshMEBtVspgLV4DkTvTxrD4yocYGknzk3z2DAuiUSLeDuyjJT/J/Zp2LsFNR7d3nOpN0AfF7FU2lJG/CefWEJCZqBDfr6Kbr5AcMvUN/spfV8LobKS/QLlo3sDyBPvanrfCRJHpCSoFlCcQdMF9Tj90IMUuQa3vUceZZ3CPSS4xfF/WEMRsQQQALuL6vgSW5vkqx53Xo5fIrOLrn3q+Arh/TLUlI2FrJEpRt3xXPLOQE7JRrk= ceph-a7981b84-025f-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-24T17:28:26.114 INFO:teuthology.orchestra.run.smithi032.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDWf4PvC0e2ZX4vQ31byvDugW7gt2B+a2H2ImNvG/O3jM0WV7pTG/P+8oINdEVDHs1JOFjdNZ+ao8GjgObqNonUX0RtfYTqVoekHG2JA2AYl8Qaj8rm8xmVuyrjizHkxvWiDw1URlh2EvUTRD7SpedPsRAcS2bYDzn99wlptBAN8G/LfNnvqa0eabAb7IA7eqHmyp5mRAI0YfB/jOTpEsI5RWpdSJY43Xvj7oIjXcuBzyCey69M57Zm4XP1LW7EE+DZ0OZsNJGEZGxRvKgshMEBtVspgLV4DkTvTxrD4yocYGknzk3z2DAuiUSLeDuyjJT/J/Zp2LsFNR7d3nOpN0AfF7FU2lJG/CefWEJCZqBDfr6Kbr5AcMvUN/spfV8LobKS/QLlo3sDyBPvanrfCRJHpCSoFlCcQdMF9Tj90IMUuQa3vUceZZ3CPSS4xfF/WEMRsQQQALuL6vgSW5vkqx53Xo5fIrOLrn3q+Arh/TLUlI2FrJEpRt3xXPLOQE7JRrk= ceph-a7981b84-025f-11ef-bc93-c7b262605968 2024-04-24T17:28:26.124 DEBUG:teuthology.orchestra.run.smithi082:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDWf4PvC0e2ZX4vQ31byvDugW7gt2B+a2H2ImNvG/O3jM0WV7pTG/P+8oINdEVDHs1JOFjdNZ+ao8GjgObqNonUX0RtfYTqVoekHG2JA2AYl8Qaj8rm8xmVuyrjizHkxvWiDw1URlh2EvUTRD7SpedPsRAcS2bYDzn99wlptBAN8G/LfNnvqa0eabAb7IA7eqHmyp5mRAI0YfB/jOTpEsI5RWpdSJY43Xvj7oIjXcuBzyCey69M57Zm4XP1LW7EE+DZ0OZsNJGEZGxRvKgshMEBtVspgLV4DkTvTxrD4yocYGknzk3z2DAuiUSLeDuyjJT/J/Zp2LsFNR7d3nOpN0AfF7FU2lJG/CefWEJCZqBDfr6Kbr5AcMvUN/spfV8LobKS/QLlo3sDyBPvanrfCRJHpCSoFlCcQdMF9Tj90IMUuQa3vUceZZ3CPSS4xfF/WEMRsQQQALuL6vgSW5vkqx53Xo5fIrOLrn3q+Arh/TLUlI2FrJEpRt3xXPLOQE7JRrk= ceph-a7981b84-025f-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-24T17:28:26.148 INFO:teuthology.orchestra.run.smithi082.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDWf4PvC0e2ZX4vQ31byvDugW7gt2B+a2H2ImNvG/O3jM0WV7pTG/P+8oINdEVDHs1JOFjdNZ+ao8GjgObqNonUX0RtfYTqVoekHG2JA2AYl8Qaj8rm8xmVuyrjizHkxvWiDw1URlh2EvUTRD7SpedPsRAcS2bYDzn99wlptBAN8G/LfNnvqa0eabAb7IA7eqHmyp5mRAI0YfB/jOTpEsI5RWpdSJY43Xvj7oIjXcuBzyCey69M57Zm4XP1LW7EE+DZ0OZsNJGEZGxRvKgshMEBtVspgLV4DkTvTxrD4yocYGknzk3z2DAuiUSLeDuyjJT/J/Zp2LsFNR7d3nOpN0AfF7FU2lJG/CefWEJCZqBDfr6Kbr5AcMvUN/spfV8LobKS/QLlo3sDyBPvanrfCRJHpCSoFlCcQdMF9Tj90IMUuQa3vUceZZ3CPSS4xfF/WEMRsQQQALuL6vgSW5vkqx53Xo5fIrOLrn3q+Arh/TLUlI2FrJEpRt3xXPLOQE7JRrk= ceph-a7981b84-025f-11ef-bc93-c7b262605968 2024-04-24T17:28:26.158 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a7981b84-025f-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-24T17:28:26.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:26 smithi032 bash[21457]: audit 2024-04-24T17:28:25.208433+0000 mon.a (mon.0) 99 : audit [INF] from='client.? v1:172.21.15.32:0/713000372' entity='client.admin' 2024-04-24T17:28:26.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:26 smithi032 bash[21457]: audit 2024-04-24T17:28:25.208433+0000 mon.a (mon.0) 99 : audit [INF] from='client.? v1:172.21.15.32:0/713000372' entity='client.admin' 2024-04-24T17:28:29.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:28 smithi032 bash[21457]: audit 2024-04-24T17:28:28.574430+0000 mon.a (mon.0) 100 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi032", "name": "osd_memory_target"}]: dispatch 2024-04-24T17:28:29.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:28 smithi032 bash[21457]: audit 2024-04-24T17:28:28.574430+0000 mon.a (mon.0) 100 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi032", "name": "osd_memory_target"}]: dispatch 2024-04-24T17:28:29.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:28 smithi032 bash[21457]: audit 2024-04-24T17:28:28.582644+0000 mon.a (mon.0) 101 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:28:29.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:28 smithi032 bash[21457]: audit 2024-04-24T17:28:28.582644+0000 mon.a (mon.0) 101 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:28:29.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:28 smithi032 bash[21457]: audit 2024-04-24T17:28:28.618051+0000 mon.a (mon.0) 102 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T17:28:29.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:28 smithi032 bash[21457]: audit 2024-04-24T17:28:28.618051+0000 mon.a (mon.0) 102 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T17:28:30.553 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T17:28:31.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.495+0000 7f78be755700 1 -- 172.21.15.32:0/1986370911 >> v1:172.21.15.32:6789/0 conn(0x7f78b8101650 legacy=0x7f78b8101a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:28:31.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.495+0000 7f78be755700 1 -- 172.21.15.32:0/1986370911 shutdown_connections 2024-04-24T17:28:31.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.495+0000 7f78be755700 1 -- 172.21.15.32:0/1986370911 >> 172.21.15.32:0/1986370911 conn(0x7f78b80fd290 msgr2=0x7f78b80ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:28:31.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.495+0000 7f78be755700 1 -- 172.21.15.32:0/1986370911 shutdown_connections 2024-04-24T17:28:31.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.495+0000 7f78be755700 1 -- 172.21.15.32:0/1986370911 wait complete. 2024-04-24T17:28:31.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.499+0000 7f78be755700 1 librados: starting msgr at 2024-04-24T17:28:31.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.499+0000 7f78be755700 1 librados: starting objecter 2024-04-24T17:28:31.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.499+0000 7f78be755700 1 Processor -- start 2024-04-24T17:28:31.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.499+0000 7f78be755700 1 -- start start 2024-04-24T17:28:31.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.499+0000 7f78be755700 1 librados: setting wanted keys 2024-04-24T17:28:31.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.499+0000 7f78be755700 1 librados: calling monclient init 2024-04-24T17:28:31.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.499+0000 7f78be755700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f78b819a120 con 0x7f78b8101650 2024-04-24T17:28:31.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.499+0000 7f78b7fff700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f78b8101650 0x7f78b8199a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:50080/0 (socket says 172.21.15.32:50080) 2024-04-24T17:28:31.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.499+0000 7f78b7fff700 1 -- 172.21.15.32:0/4162211384 learned_addr learned my addr 172.21.15.32:0/4162211384 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:28:31.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.499+0000 7f78b57fa700 10 client.?.objecter ms_handle_connect 0x7f78b8101650 2024-04-24T17:28:31.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.499+0000 7f78b57fa700 10 client.?.objecter resend_mon_ops 2024-04-24T17:28:31.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.499+0000 7f78b57fa700 1 -- 172.21.15.32:0/4162211384 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4110174480 0 0) 0x7f78b819a120 con 0x7f78b8101650 2024-04-24T17:28:31.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.499+0000 7f78b57fa700 1 -- 172.21.15.32:0/4162211384 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78a00056e0 con 0x7f78b8101650 2024-04-24T17:28:31.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.499+0000 7f78b57fa700 1 -- 172.21.15.32:0/4162211384 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2186588888 0 0) 0x7f78b8103190 con 0x7f78b8101650 2024-04-24T17:28:31.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.499+0000 7f78b57fa700 1 -- 172.21.15.32:0/4162211384 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f78b819a120 con 0x7f78b8101650 2024-04-24T17:28:31.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.499+0000 7f78b57fa700 1 -- 172.21.15.32:0/4162211384 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f78a80044c0 con 0x7f78b8101650 2024-04-24T17:28:31.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.499+0000 7f78b57fa700 1 -- 172.21.15.32:0/4162211384 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3874791392 0 0) 0x7f78b819a120 con 0x7f78b8101650 2024-04-24T17:28:31.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.499+0000 7f78b57fa700 1 -- 172.21.15.32:0/4162211384 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78b819a2f0 con 0x7f78b8101650 2024-04-24T17:28:31.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.499+0000 7f78be755700 1 -- 172.21.15.32:0/4162211384 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f78b819a5e0 con 0x7f78b8101650 2024-04-24T17:28:31.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.499+0000 7f78be755700 10 client.14174.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-24T17:28:31.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.499+0000 7f78be755700 1 -- 172.21.15.32:0/4162211384 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f78b819e130 con 0x7f78b8101650 2024-04-24T17:28:31.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.499+0000 7f78be755700 1 librados: init done 2024-04-24T17:28:31.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.499+0000 7f78b57fa700 1 -- 172.21.15.32:0/4162211384 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(25 keys) ==== 1042+0+0 (unknown 2768897130 0 0) 0x7f78a80016a0 con 0x7f78b8101650 2024-04-24T17:28:31.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.499+0000 7f78b57fa700 1 -- 172.21.15.32:0/4162211384 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f78a8004e00 con 0x7f78b8101650 2024-04-24T17:28:31.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.499+0000 7f78b57fa700 1 -- 172.21.15.32:0/4162211384 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 13) ==== 46579+0+0 (unknown 583022566 0 0) 0x7f78a8004fc0 con 0x7f78b8101650 2024-04-24T17:28:31.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.499+0000 7f78be755700 1 -- 172.21.15.32:0/4162211384 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78a4005150 con 0x7f78b8101650 2024-04-24T17:28:31.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.499+0000 7f78b57fa700 1 -- 172.21.15.32:0/4162211384 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 1575289653 0 0) 0x7f78a80481c0 con 0x7f78b8101650 2024-04-24T17:28:31.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.499+0000 7f78b57fa700 10 client.14174.objecter ms_dispatch 0x7f78b8001550 osd_map(3..3 src has 1..3) 2024-04-24T17:28:31.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.499+0000 7f78b57fa700 3 client.14174.objecter handle_osd_map got epochs [3,3] > 0 2024-04-24T17:28:31.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.499+0000 7f78b57fa700 3 client.14174.objecter handle_osd_map decoding full epoch 3 2024-04-24T17:28:31.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.499+0000 7f78b57fa700 20 client.14174.objecter dump_active .. 0 homeless 2024-04-24T17:28:31.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.499+0000 7f78b57fa700 10 client.14174.objecter ms_handle_connect 0x7f78a003c2a0 2024-04-24T17:28:31.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.507+0000 7f78b57fa700 1 -- 172.21.15.32:0/4162211384 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188728 (unknown 1092875540 0 2656817166) 0x7f78a00056e0 con 0x7f78b8101650 2024-04-24T17:28:31.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.719+0000 7f78be755700 1 -- 172.21.15.32:0/4162211384 --> v1:172.21.15.32:6789/0 -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f78a4005440 con 0x7f78b8101650 2024-04-24T17:28:31.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.723+0000 7f78b57fa700 1 -- 172.21.15.32:0/4162211384 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (unknown 3028693289 0 0) 0x7f78a8017720 con 0x7f78b8101650 2024-04-24T17:28:31.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.727+0000 7f78be755700 10 librados: watch_flush enter 2024-04-24T17:28:31.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.727+0000 7f78be755700 10 librados: watch_flush exit 2024-04-24T17:28:31.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.727+0000 7f78be755700 20 client.14174.objecter shutdown clearing up homeless session... 2024-04-24T17:28:31.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.727+0000 7f78be755700 10 client.14174.objecter successfully canceled tick 2024-04-24T17:28:31.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.727+0000 7f78be755700 1 -- 172.21.15.32:0/4162211384 >> v1:172.21.15.32:6800/1847389373 conn(0x7f78a003c2a0 legacy=0x7f78a003e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:28:31.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.727+0000 7f78be755700 1 -- 172.21.15.32:0/4162211384 >> v1:172.21.15.32:6789/0 conn(0x7f78b8101650 legacy=0x7f78b8199a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:28:31.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.727+0000 7f78be755700 1 -- 172.21.15.32:0/4162211384 shutdown_connections 2024-04-24T17:28:31.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.727+0000 7f78be755700 1 -- 172.21.15.32:0/4162211384 >> 172.21.15.32:0/4162211384 conn(0x7f78b80fd290 msgr2=0x7f78b80ff680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:28:31.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.727+0000 7f78be755700 1 -- 172.21.15.32:0/4162211384 shutdown_connections 2024-04-24T17:28:31.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.727+0000 7f78be755700 1 -- 172.21.15.32:0/4162211384 wait complete. 2024-04-24T17:28:31.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:31.727+0000 7f78be755700 1 librados: shutdown 2024-04-24T17:28:32.387 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-24T17:28:32.387 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a7981b84-025f-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-24T17:28:33.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:32 smithi032 bash[21457]: audit 2024-04-24T17:28:31.730172+0000 mon.a (mon.0) 103 : audit [INF] from='client.? v1:172.21.15.32:0/4162211384' entity='client.admin' 2024-04-24T17:28:33.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:32 smithi032 bash[21457]: audit 2024-04-24T17:28:31.730172+0000 mon.a (mon.0) 103 : audit [INF] from='client.? v1:172.21.15.32:0/4162211384' entity='client.admin' 2024-04-24T17:28:34.752 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:34 smithi032 bash[21457]: cluster 2024-04-24T17:28:34.629151+0000 mon.a (mon.0) 104 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED) 2024-04-24T17:28:34.752 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:34 smithi032 bash[21457]: cluster 2024-04-24T17:28:34.629151+0000 mon.a (mon.0) 104 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED) 2024-04-24T17:28:36.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:35 smithi032 bash[21457]: cluster 2024-04-24T17:28:34.577913+0000 mgr.y (mgr.14152) 9 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:28:36.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:35 smithi032 bash[21457]: cluster 2024-04-24T17:28:34.577913+0000 mgr.y (mgr.14152) 9 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:28:36.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:35 smithi032 bash[21457]: cluster 2024-04-24T17:28:34.751450+0000 mgr.y (mgr.14152) 10 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:28:36.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:35 smithi032 bash[21457]: cluster 2024-04-24T17:28:34.751450+0000 mgr.y (mgr.14152) 10 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:28:36.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:35 smithi032 bash[21457]: cluster 2024-04-24T17:28:34.752084+0000 mgr.y (mgr.14152) 11 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:28:36.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:35 smithi032 bash[21457]: cluster 2024-04-24T17:28:34.752084+0000 mgr.y (mgr.14152) 11 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:28:36.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:35 smithi032 bash[21457]: audit 2024-04-24T17:28:34.756803+0000 mon.a (mon.0) 105 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:28:36.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:35 smithi032 bash[21457]: audit 2024-04-24T17:28:34.756803+0000 mon.a (mon.0) 105 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:28:36.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:35 smithi032 bash[21457]: audit 2024-04-24T17:28:34.792391+0000 mon.a (mon.0) 106 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T17:28:36.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:35 smithi032 bash[21457]: audit 2024-04-24T17:28:34.792391+0000 mon.a (mon.0) 106 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T17:28:36.872 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T17:28:37.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.759+0000 7f28f416f700 1 -- 172.21.15.32:0/3465455158 >> v1:172.21.15.32:6789/0 conn(0x7f28ec102db0 legacy=0x7f28ec1031d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:28:37.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.759+0000 7f28f416f700 1 -- 172.21.15.32:0/3465455158 shutdown_connections 2024-04-24T17:28:37.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.759+0000 7f28f416f700 1 -- 172.21.15.32:0/3465455158 >> 172.21.15.32:0/3465455158 conn(0x7f28ec0fe370 msgr2=0x7f28ec100790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:28:37.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.759+0000 7f28f416f700 1 -- 172.21.15.32:0/3465455158 shutdown_connections 2024-04-24T17:28:37.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.759+0000 7f28f416f700 1 -- 172.21.15.32:0/3465455158 wait complete. 2024-04-24T17:28:37.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.759+0000 7f28f416f700 1 librados: starting msgr at 2024-04-24T17:28:37.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.759+0000 7f28f416f700 1 librados: starting objecter 2024-04-24T17:28:37.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.759+0000 7f28f416f700 1 Processor -- start 2024-04-24T17:28:37.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.759+0000 7f28f416f700 1 -- start start 2024-04-24T17:28:37.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.759+0000 7f28f416f700 1 librados: setting wanted keys 2024-04-24T17:28:37.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.759+0000 7f28f416f700 1 librados: calling monclient init 2024-04-24T17:28:37.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.759+0000 7f28f416f700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f28ec197f00 con 0x7f28ec102db0 2024-04-24T17:28:37.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.759+0000 7f28f1f0b700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f28ec102db0 0x7f28ec1977f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:42240/0 (socket says 172.21.15.32:42240) 2024-04-24T17:28:37.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.759+0000 7f28f1f0b700 1 -- 172.21.15.32:0/409328629 learned_addr learned my addr 172.21.15.32:0/409328629 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:28:37.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.759+0000 7f28eaffd700 10 client.?.objecter ms_handle_connect 0x7f28ec102db0 2024-04-24T17:28:37.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.759+0000 7f28eaffd700 10 client.?.objecter resend_mon_ops 2024-04-24T17:28:37.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.759+0000 7f28eaffd700 1 -- 172.21.15.32:0/409328629 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2534726321 0 0) 0x7f28ec197f00 con 0x7f28ec102db0 2024-04-24T17:28:37.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.759+0000 7f28eaffd700 1 -- 172.21.15.32:0/409328629 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f28d80035b0 con 0x7f28ec102db0 2024-04-24T17:28:37.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.759+0000 7f28eaffd700 1 -- 172.21.15.32:0/409328629 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1044749104 0 0) 0x7f28d80035b0 con 0x7f28ec102db0 2024-04-24T17:28:37.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.759+0000 7f28eaffd700 1 -- 172.21.15.32:0/409328629 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f28ec197f00 con 0x7f28ec102db0 2024-04-24T17:28:37.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.759+0000 7f28eaffd700 1 -- 172.21.15.32:0/409328629 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f28d40016f0 con 0x7f28ec102db0 2024-04-24T17:28:37.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.759+0000 7f28eaffd700 1 -- 172.21.15.32:0/409328629 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4020376515 0 0) 0x7f28ec197f00 con 0x7f28ec102db0 2024-04-24T17:28:37.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.759+0000 7f28eaffd700 1 -- 172.21.15.32:0/409328629 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28ec1980d0 con 0x7f28ec102db0 2024-04-24T17:28:37.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.759+0000 7f28f416f700 1 -- 172.21.15.32:0/409328629 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f28ec1983c0 con 0x7f28ec102db0 2024-04-24T17:28:37.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.759+0000 7f28f416f700 10 client.14176.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-24T17:28:37.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.759+0000 7f28f416f700 1 -- 172.21.15.32:0/409328629 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f28ec19bf10 con 0x7f28ec102db0 2024-04-24T17:28:37.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.759+0000 7f28f416f700 1 librados: init done 2024-04-24T17:28:37.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.759+0000 7f28eaffd700 1 -- 172.21.15.32:0/409328629 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(25 keys) ==== 1042+0+0 (unknown 2768897130 0 0) 0x7f28d4001510 con 0x7f28ec102db0 2024-04-24T17:28:37.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.759+0000 7f28eaffd700 1 -- 172.21.15.32:0/409328629 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f28d4004f40 con 0x7f28ec102db0 2024-04-24T17:28:37.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.763+0000 7f28eaffd700 1 -- 172.21.15.32:0/409328629 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 13) ==== 46579+0+0 (unknown 583022566 0 0) 0x7f28d4005180 con 0x7f28ec102db0 2024-04-24T17:28:37.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.763+0000 7f28f416f700 1 -- 172.21.15.32:0/409328629 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28cc005150 con 0x7f28ec102db0 2024-04-24T17:28:37.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.763+0000 7f28eaffd700 1 -- 172.21.15.32:0/409328629 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 1575289653 0 0) 0x7f28d4048380 con 0x7f28ec102db0 2024-04-24T17:28:37.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.763+0000 7f28eaffd700 10 client.14176.objecter ms_dispatch 0x7f28ec001550 osd_map(3..3 src has 1..3) 2024-04-24T17:28:37.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.763+0000 7f28eaffd700 3 client.14176.objecter handle_osd_map got epochs [3,3] > 0 2024-04-24T17:28:37.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.763+0000 7f28eaffd700 3 client.14176.objecter handle_osd_map decoding full epoch 3 2024-04-24T17:28:37.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.763+0000 7f28eaffd700 20 client.14176.objecter dump_active .. 0 homeless 2024-04-24T17:28:37.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.763+0000 7f28eaffd700 10 client.14176.objecter ms_handle_connect 0x7f28d803c340 2024-04-24T17:28:37.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.767+0000 7f28eaffd700 1 -- 172.21.15.32:0/409328629 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188728 (unknown 1092875540 0 2656817166) 0x7f28d4017940 con 0x7f28ec102db0 2024-04-24T17:28:37.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.983+0000 7f28f416f700 1 -- 172.21.15.32:0/409328629 --> v1:172.21.15.32:6800/1847389373 -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f28cc000bc0 con 0x7f28d803c340 2024-04-24T17:28:37.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.987+0000 7f28eaffd700 1 -- 172.21.15.32:0/409328629 <== mgr.14152 v1:172.21.15.32:6800/1847389373 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (unknown 0 0 0) 0x7f28cc000bc0 con 0x7f28d803c340 2024-04-24T17:28:37.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.987+0000 7f28f416f700 10 librados: watch_flush enter 2024-04-24T17:28:37.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.987+0000 7f28f416f700 10 librados: watch_flush exit 2024-04-24T17:28:37.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.987+0000 7f28f416f700 20 client.14176.objecter shutdown clearing up homeless session... 2024-04-24T17:28:37.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.987+0000 7f28f416f700 10 client.14176.objecter successfully canceled tick 2024-04-24T17:28:37.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.987+0000 7f28f416f700 1 -- 172.21.15.32:0/409328629 >> v1:172.21.15.32:6800/1847389373 conn(0x7f28d803c340 legacy=0x7f28d803e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:28:37.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.987+0000 7f28f416f700 1 -- 172.21.15.32:0/409328629 >> v1:172.21.15.32:6789/0 conn(0x7f28ec102db0 legacy=0x7f28ec1977f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:28:37.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.987+0000 7f28f416f700 1 -- 172.21.15.32:0/409328629 shutdown_connections 2024-04-24T17:28:37.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.987+0000 7f28f416f700 1 -- 172.21.15.32:0/409328629 >> 172.21.15.32:0/409328629 conn(0x7f28ec0fe370 msgr2=0x7f28ec100760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:28:37.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.991+0000 7f28f416f700 1 -- 172.21.15.32:0/409328629 shutdown_connections 2024-04-24T17:28:37.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.991+0000 7f28f416f700 1 -- 172.21.15.32:0/409328629 wait complete. 2024-04-24T17:28:37.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:37.991+0000 7f28f416f700 1 librados: shutdown 2024-04-24T17:28:38.001 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:37 smithi032 bash[21457]: cluster 2024-04-24T17:28:36.752854+0000 mgr.y (mgr.14152) 12 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:28:38.001 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:37 smithi032 bash[21457]: cluster 2024-04-24T17:28:36.752854+0000 mgr.y (mgr.14152) 12 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:28:38.637 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi082 2024-04-24T17:28:38.637 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-04-24T17:28:38.637 DEBUG:teuthology.orchestra.run.smithi082:> dd of=/etc/ceph/ceph.conf 2024-04-24T17:28:38.643 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-04-24T17:28:38.644 DEBUG:teuthology.orchestra.run.smithi082:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-24T17:28:38.692 INFO:tasks.cephadm:Adding host smithi082 to orchestrator... 2024-04-24T17:28:38.692 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a7981b84-025f-11ef-bc93-c7b262605968 -- ceph orch host add smithi082 2024-04-24T17:28:39.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:38 smithi032 bash[21457]: audit 2024-04-24T17:28:37.988010+0000 mgr.y (mgr.14152) 13 : audit [DBG] from='client.14176 v1:172.21.15.32:0/409328629' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T17:28:39.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:38 smithi032 bash[21457]: audit 2024-04-24T17:28:37.988010+0000 mgr.y (mgr.14152) 13 : audit [DBG] from='client.14176 v1:172.21.15.32:0/409328629' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T17:28:39.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:38 smithi032 bash[21457]: audit 2024-04-24T17:28:37.991676+0000 mon.a (mon.0) 107 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:28:39.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:38 smithi032 bash[21457]: audit 2024-04-24T17:28:37.991676+0000 mon.a (mon.0) 107 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:28:40.176 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:39 smithi032 bash[21457]: cluster 2024-04-24T17:28:38.753391+0000 mgr.y (mgr.14152) 14 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:28:40.176 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:39 smithi032 bash[21457]: cluster 2024-04-24T17:28:38.753391+0000 mgr.y (mgr.14152) 14 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:28:41.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:40 smithi032 bash[21457]: audit 2024-04-24T17:28:40.977324+0000 mon.a (mon.0) 108 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:28:41.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:40 smithi032 bash[21457]: audit 2024-04-24T17:28:40.977324+0000 mon.a (mon.0) 108 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:28:41.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:40 smithi032 bash[21457]: audit 2024-04-24T17:28:40.979224+0000 mon.a (mon.0) 109 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T17:28:41.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:40 smithi032 bash[21457]: audit 2024-04-24T17:28:40.979224+0000 mon.a (mon.0) 109 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T17:28:42.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:42 smithi032 bash[21457]: cluster 2024-04-24T17:28:40.754045+0000 mgr.y (mgr.14152) 15 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:28:42.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:42 smithi032 bash[21457]: cluster 2024-04-24T17:28:40.754045+0000 mgr.y (mgr.14152) 15 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:28:42.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:42 smithi032 bash[21457]: cephadm 2024-04-24T17:28:40.981988+0000 mgr.y (mgr.14152) 16 : cephadm [INF] Updating smithi032:/etc/ceph/ceph.conf 2024-04-24T17:28:42.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:42 smithi032 bash[21457]: cephadm 2024-04-24T17:28:40.981988+0000 mgr.y (mgr.14152) 16 : cephadm [INF] Updating smithi032:/etc/ceph/ceph.conf 2024-04-24T17:28:42.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:42 smithi032 bash[21457]: cephadm 2024-04-24T17:28:41.198285+0000 mgr.y (mgr.14152) 17 : cephadm [INF] Updating smithi032:/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T17:28:42.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:42 smithi032 bash[21457]: cephadm 2024-04-24T17:28:41.198285+0000 mgr.y (mgr.14152) 17 : cephadm [INF] Updating smithi032:/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T17:28:42.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:42 smithi032 bash[21457]: cephadm 2024-04-24T17:28:41.312660+0000 mgr.y (mgr.14152) 18 : cephadm [INF] Updating smithi032:/etc/ceph/ceph.client.admin.keyring 2024-04-24T17:28:42.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:42 smithi032 bash[21457]: cephadm 2024-04-24T17:28:41.312660+0000 mgr.y (mgr.14152) 18 : cephadm [INF] Updating smithi032:/etc/ceph/ceph.client.admin.keyring 2024-04-24T17:28:42.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:42 smithi032 bash[21457]: audit 2024-04-24T17:28:41.545473+0000 mon.a (mon.0) 110 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:28:42.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:42 smithi032 bash[21457]: audit 2024-04-24T17:28:41.545473+0000 mon.a (mon.0) 110 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:28:42.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:42 smithi032 bash[21457]: audit 2024-04-24T17:28:41.549104+0000 mon.a (mon.0) 111 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:28:42.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:42 smithi032 bash[21457]: audit 2024-04-24T17:28:41.549104+0000 mon.a (mon.0) 111 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:28:42.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:42 smithi032 bash[21457]: audit 2024-04-24T17:28:41.582459+0000 mon.a (mon.0) 112 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T17:28:42.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:42 smithi032 bash[21457]: audit 2024-04-24T17:28:41.582459+0000 mon.a (mon.0) 112 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T17:28:43.108 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T17:28:43.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:43 smithi032 bash[21457]: cephadm 2024-04-24T17:28:41.426976+0000 mgr.y (mgr.14152) 19 : cephadm [INF] Updating smithi032:/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T17:28:43.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:43 smithi032 bash[21457]: cephadm 2024-04-24T17:28:41.426976+0000 mgr.y (mgr.14152) 19 : cephadm [INF] Updating smithi032:/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T17:28:43.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:43 smithi032 bash[21457]: cluster 2024-04-24T17:28:41.546599+0000 mgr.y (mgr.14152) 20 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:28:43.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:43 smithi032 bash[21457]: cluster 2024-04-24T17:28:41.546599+0000 mgr.y (mgr.14152) 20 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:28:43.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:43 smithi032 bash[21457]: cluster 2024-04-24T17:28:41.547200+0000 mgr.y (mgr.14152) 21 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:28:43.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:43 smithi032 bash[21457]: cluster 2024-04-24T17:28:41.547200+0000 mgr.y (mgr.14152) 21 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:28:44.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:44.047+0000 7ff451822700 1 -- 172.21.15.32:0/2221944625 >> v1:172.21.15.32:6789/0 conn(0x7ff44c1051d0 legacy=0x7ff44c1055b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:28:44.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:44.047+0000 7ff451822700 1 -- 172.21.15.32:0/2221944625 shutdown_connections 2024-04-24T17:28:44.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:44.047+0000 7ff451822700 1 -- 172.21.15.32:0/2221944625 >> 172.21.15.32:0/2221944625 conn(0x7ff44c100aa0 msgr2=0x7ff44c102ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:28:44.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:44.047+0000 7ff451822700 1 -- 172.21.15.32:0/2221944625 shutdown_connections 2024-04-24T17:28:44.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:44.047+0000 7ff451822700 1 -- 172.21.15.32:0/2221944625 wait complete. 2024-04-24T17:28:44.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:44.051+0000 7ff451822700 1 librados: starting msgr at 2024-04-24T17:28:44.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:44.051+0000 7ff451822700 1 librados: starting objecter 2024-04-24T17:28:44.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:44.051+0000 7ff451822700 1 Processor -- start 2024-04-24T17:28:44.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:44.051+0000 7ff451822700 1 -- start start 2024-04-24T17:28:44.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:44.051+0000 7ff451822700 1 librados: setting wanted keys 2024-04-24T17:28:44.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:44.051+0000 7ff451822700 1 librados: calling monclient init 2024-04-24T17:28:44.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:44.051+0000 7ff451822700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff44c19a450 con 0x7ff44c1051d0 2024-04-24T17:28:44.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:44.051+0000 7ff44affd700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff44c1051d0 0x7ff44c199d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:42252/0 (socket says 172.21.15.32:42252) 2024-04-24T17:28:44.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:44.051+0000 7ff44affd700 1 -- 172.21.15.32:0/3686357677 learned_addr learned my addr 172.21.15.32:0/3686357677 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:28:44.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:44.051+0000 7ff450820700 10 client.?.objecter ms_handle_connect 0x7ff44c1051d0 2024-04-24T17:28:44.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:44.051+0000 7ff450820700 10 client.?.objecter resend_mon_ops 2024-04-24T17:28:44.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:44.051+0000 7ff450820700 1 -- 172.21.15.32:0/3686357677 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1818308164 0 0) 0x7ff44c19a450 con 0x7ff44c1051d0 2024-04-24T17:28:44.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:44.051+0000 7ff450820700 1 -- 172.21.15.32:0/3686357677 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff4340056e0 con 0x7ff44c1051d0 2024-04-24T17:28:44.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:44.051+0000 7ff450820700 1 -- 172.21.15.32:0/3686357677 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 196616917 0 0) 0x7ff4340056e0 con 0x7ff44c1051d0 2024-04-24T17:28:44.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:44.051+0000 7ff450820700 1 -- 172.21.15.32:0/3686357677 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff44c19a450 con 0x7ff44c1051d0 2024-04-24T17:28:44.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:44.051+0000 7ff450820700 1 -- 172.21.15.32:0/3686357677 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7ff43c004630 con 0x7ff44c1051d0 2024-04-24T17:28:44.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:44.051+0000 7ff450820700 1 -- 172.21.15.32:0/3686357677 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2860636942 0 0) 0x7ff44c19a450 con 0x7ff44c1051d0 2024-04-24T17:28:44.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:44.051+0000 7ff450820700 1 -- 172.21.15.32:0/3686357677 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff44c19a620 con 0x7ff44c1051d0 2024-04-24T17:28:44.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:44.051+0000 7ff451822700 1 -- 172.21.15.32:0/3686357677 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff44c19a910 con 0x7ff44c1051d0 2024-04-24T17:28:44.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:44.051+0000 7ff451822700 10 client.14178.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-24T17:28:44.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:44.051+0000 7ff451822700 1 -- 172.21.15.32:0/3686357677 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff44c1a2ad0 con 0x7ff44c1051d0 2024-04-24T17:28:44.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:44.051+0000 7ff451822700 1 librados: init done 2024-04-24T17:28:44.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:44.051+0000 7ff450820700 1 -- 172.21.15.32:0/3686357677 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(25 keys) ==== 1042+0+0 (unknown 2768897130 0 0) 0x7ff43c001d10 con 0x7ff44c1051d0 2024-04-24T17:28:44.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:44.051+0000 7ff450820700 1 -- 172.21.15.32:0/3686357677 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7ff43c004c40 con 0x7ff44c1051d0 2024-04-24T17:28:44.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:44.051+0000 7ff450820700 1 -- 172.21.15.32:0/3686357677 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 13) ==== 46579+0+0 (unknown 583022566 0 0) 0x7ff43c004ec0 con 0x7ff44c1051d0 2024-04-24T17:28:44.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:44.051+0000 7ff451822700 1 -- 172.21.15.32:0/3686357677 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff438005120 con 0x7ff44c1051d0 2024-04-24T17:28:44.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:44.051+0000 7ff450820700 1 -- 172.21.15.32:0/3686357677 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 1575289653 0 0) 0x7ff43c047f60 con 0x7ff44c1051d0 2024-04-24T17:28:44.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:44.051+0000 7ff450820700 10 client.14178.objecter ms_dispatch 0x7ff44c001550 osd_map(3..3 src has 1..3) 2024-04-24T17:28:44.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:44.051+0000 7ff450820700 3 client.14178.objecter handle_osd_map got epochs [3,3] > 0 2024-04-24T17:28:44.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:44.051+0000 7ff450820700 3 client.14178.objecter handle_osd_map decoding full epoch 3 2024-04-24T17:28:44.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:44.051+0000 7ff450820700 20 client.14178.objecter dump_active .. 0 homeless 2024-04-24T17:28:44.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:44.051+0000 7ff450820700 10 client.14178.objecter ms_handle_connect 0x7ff43403c2f0 2024-04-24T17:28:44.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:44.059+0000 7ff450820700 1 -- 172.21.15.32:0/3686357677 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188728 (unknown 1092875540 0 2656817166) 0x7ff43c0175c0 con 0x7ff44c1051d0 2024-04-24T17:28:44.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:44.275+0000 7ff451822700 1 -- 172.21.15.32:0/3686357677 --> v1:172.21.15.32:6800/1847389373 -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi082", "target": ["mon-mgr", ""]}) -- 0x7ff438000b90 con 0x7ff43403c2f0 2024-04-24T17:28:45.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:45 smithi032 bash[21457]: cluster 2024-04-24T17:28:43.547960+0000 mgr.y (mgr.14152) 22 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:28:45.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:45 smithi032 bash[21457]: cluster 2024-04-24T17:28:43.547960+0000 mgr.y (mgr.14152) 22 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:28:45.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:45 smithi032 bash[21457]: audit 2024-04-24T17:28:44.280687+0000 mgr.y (mgr.14152) 23 : audit [DBG] from='client.14178 v1:172.21.15.32:0/3686357677' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi082", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T17:28:45.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:45 smithi032 bash[21457]: audit 2024-04-24T17:28:44.280687+0000 mgr.y (mgr.14152) 23 : audit [DBG] from='client.14178 v1:172.21.15.32:0/3686357677' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi082", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T17:28:46.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:46 smithi032 bash[21457]: cephadm 2024-04-24T17:28:44.996769+0000 mgr.y (mgr.14152) 24 : cephadm [INF] Deploying cephadm binary to smithi082 2024-04-24T17:28:46.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:46 smithi032 bash[21457]: cephadm 2024-04-24T17:28:44.996769+0000 mgr.y (mgr.14152) 24 : cephadm [INF] Deploying cephadm binary to smithi082 2024-04-24T17:28:47.333 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:47 smithi032 bash[21457]: cluster 2024-04-24T17:28:45.548691+0000 mgr.y (mgr.14152) 25 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:28:47.333 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:47 smithi032 bash[21457]: cluster 2024-04-24T17:28:45.548691+0000 mgr.y (mgr.14152) 25 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:28:49.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:49.055+0000 7ff451021700 10 client.14178.objecter tick 2024-04-24T17:28:49.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:49 smithi032 bash[21457]: cluster 2024-04-24T17:28:47.549433+0000 mgr.y (mgr.14152) 26 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:28:49.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:49 smithi032 bash[21457]: cluster 2024-04-24T17:28:47.549433+0000 mgr.y (mgr.14152) 26 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:28:50.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:50.474+0000 7ff450820700 1 -- 172.21.15.32:0/3686357677 <== mgr.14152 v1:172.21.15.32:6800/1847389373 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (unknown 0 0 3989977768) 0x7ff438000b90 con 0x7ff43403c2f0 2024-04-24T17:28:50.481 INFO:teuthology.orchestra.run.smithi032.stdout:Added host 'smithi082' with addr '172.21.15.82' 2024-04-24T17:28:50.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:50.478+0000 7ff451822700 10 librados: watch_flush enter 2024-04-24T17:28:50.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:50.478+0000 7ff451822700 10 librados: watch_flush exit 2024-04-24T17:28:50.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:50.478+0000 7ff451822700 20 client.14178.objecter shutdown clearing up homeless session... 2024-04-24T17:28:50.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:50.478+0000 7ff451822700 10 client.14178.objecter successfully canceled tick 2024-04-24T17:28:50.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:50.478+0000 7ff451822700 1 -- 172.21.15.32:0/3686357677 >> v1:172.21.15.32:6800/1847389373 conn(0x7ff43403c2f0 legacy=0x7ff43403e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:28:50.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:50.478+0000 7ff451822700 1 -- 172.21.15.32:0/3686357677 >> v1:172.21.15.32:6789/0 conn(0x7ff44c1051d0 legacy=0x7ff44c199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:28:50.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:50.478+0000 7ff451822700 1 -- 172.21.15.32:0/3686357677 shutdown_connections 2024-04-24T17:28:50.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:50.478+0000 7ff451822700 1 -- 172.21.15.32:0/3686357677 >> 172.21.15.32:0/3686357677 conn(0x7ff44c100aa0 msgr2=0x7ff44c10c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:28:50.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:50.478+0000 7ff451822700 1 -- 172.21.15.32:0/3686357677 shutdown_connections 2024-04-24T17:28:50.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:50.478+0000 7ff451822700 1 -- 172.21.15.32:0/3686357677 wait complete. 2024-04-24T17:28:50.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:50.478+0000 7ff451822700 1 librados: shutdown 2024-04-24T17:28:50.500 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:50 smithi032 bash[21457]: audit 2024-04-24T17:28:49.360105+0000 mon.a (mon.0) 113 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:28:50.500 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:50 smithi032 bash[21457]: audit 2024-04-24T17:28:49.360105+0000 mon.a (mon.0) 113 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:28:50.500 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:50 smithi032 bash[21457]: audit 2024-04-24T17:28:49.383711+0000 mon.a (mon.0) 114 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T17:28:50.500 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:50 smithi032 bash[21457]: audit 2024-04-24T17:28:49.383711+0000 mon.a (mon.0) 114 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T17:28:50.501 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:50 smithi032 bash[21457]: audit 2024-04-24T17:28:49.394534+0000 mon.a (mon.0) 115 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:28:50.501 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:50 smithi032 bash[21457]: audit 2024-04-24T17:28:49.394534+0000 mon.a (mon.0) 115 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:28:51.129 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a7981b84-025f-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-24T17:28:51.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:51 smithi032 bash[21457]: cluster 2024-04-24T17:28:49.389199+0000 mgr.y (mgr.14152) 27 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:28:51.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:51 smithi032 bash[21457]: cluster 2024-04-24T17:28:49.389199+0000 mgr.y (mgr.14152) 27 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:28:51.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:51 smithi032 bash[21457]: cluster 2024-04-24T17:28:49.389894+0000 mgr.y (mgr.14152) 28 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:28:51.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:51 smithi032 bash[21457]: cluster 2024-04-24T17:28:49.389894+0000 mgr.y (mgr.14152) 28 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:28:51.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:51 smithi032 bash[21457]: audit 2024-04-24T17:28:50.480686+0000 mon.a (mon.0) 116 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:28:51.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:51 smithi032 bash[21457]: audit 2024-04-24T17:28:50.480686+0000 mon.a (mon.0) 116 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:28:51.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:51 smithi032 bash[21457]: audit 2024-04-24T17:28:50.516262+0000 mon.a (mon.0) 117 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T17:28:51.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:51 smithi032 bash[21457]: audit 2024-04-24T17:28:50.516262+0000 mon.a (mon.0) 117 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T17:28:51.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:51 smithi032 bash[21457]: audit 2024-04-24T17:28:50.987627+0000 mon.a (mon.0) 118 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:28:51.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:51 smithi032 bash[21457]: audit 2024-04-24T17:28:50.987627+0000 mon.a (mon.0) 118 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:28:52.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:52 smithi032 bash[21457]: cephadm 2024-04-24T17:28:50.481597+0000 mgr.y (mgr.14152) 29 : cephadm [INF] Added host smithi082 2024-04-24T17:28:52.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:52 smithi032 bash[21457]: cephadm 2024-04-24T17:28:50.481597+0000 mgr.y (mgr.14152) 29 : cephadm [INF] Added host smithi082 2024-04-24T17:28:53.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:53 smithi032 bash[21457]: cluster 2024-04-24T17:28:51.390558+0000 mgr.y (mgr.14152) 30 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:28:53.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:53 smithi032 bash[21457]: cluster 2024-04-24T17:28:51.390558+0000 mgr.y (mgr.14152) 30 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:28:53.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:53 smithi032 bash[21457]: audit 2024-04-24T17:28:52.461457+0000 mon.a (mon.0) 119 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:28:53.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:53 smithi032 bash[21457]: audit 2024-04-24T17:28:52.461457+0000 mon.a (mon.0) 119 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:28:55.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:54 smithi032 bash[21457]: cluster 2024-04-24T17:28:53.391312+0000 mgr.y (mgr.14152) 31 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:28:55.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:54 smithi032 bash[21457]: cluster 2024-04-24T17:28:53.391312+0000 mgr.y (mgr.14152) 31 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:28:55.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:54 smithi032 bash[21457]: audit 2024-04-24T17:28:53.603070+0000 mon.a (mon.0) 120 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:28:55.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:54 smithi032 bash[21457]: audit 2024-04-24T17:28:53.603070+0000 mon.a (mon.0) 120 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:28:55.454 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T17:28:56.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:56.946+0000 7fbf7ad07700 1 -- 172.21.15.32:0/48876371 >> v1:172.21.15.32:6789/0 conn(0x7fbf74101b80 legacy=0x7fbf74101f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:28:56.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:56.946+0000 7fbf7ad07700 1 -- 172.21.15.32:0/48876371 shutdown_connections 2024-04-24T17:28:56.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:56.946+0000 7fbf7ad07700 1 -- 172.21.15.32:0/48876371 >> 172.21.15.32:0/48876371 conn(0x7fbf740fd7c0 msgr2=0x7fbf740ffbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:28:56.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:56.946+0000 7fbf7ad07700 1 -- 172.21.15.32:0/48876371 shutdown_connections 2024-04-24T17:28:56.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:56.946+0000 7fbf7ad07700 1 -- 172.21.15.32:0/48876371 wait complete. 2024-04-24T17:28:56.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:56.950+0000 7fbf7ad07700 1 librados: starting msgr at 2024-04-24T17:28:56.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:56.950+0000 7fbf7ad07700 1 librados: starting objecter 2024-04-24T17:28:56.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:56.950+0000 7fbf7ad07700 1 Processor -- start 2024-04-24T17:28:56.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:56.950+0000 7fbf7ad07700 1 -- start start 2024-04-24T17:28:56.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:56.950+0000 7fbf7ad07700 1 librados: setting wanted keys 2024-04-24T17:28:56.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:56.950+0000 7fbf7ad07700 1 librados: calling monclient init 2024-04-24T17:28:56.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:56.950+0000 7fbf7ad07700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbf7419a990 con 0x7fbf74199ea0 2024-04-24T17:28:56.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:56.950+0000 7fbf78aa3700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fbf74199ea0 0x7fbf7419a280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51060/0 (socket says 172.21.15.32:51060) 2024-04-24T17:28:56.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:56.950+0000 7fbf78aa3700 1 -- 172.21.15.32:0/3175493670 learned_addr learned my addr 172.21.15.32:0/3175493670 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:28:56.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:56.950+0000 7fbf71ffb700 10 client.?.objecter ms_handle_connect 0x7fbf74199ea0 2024-04-24T17:28:56.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:56.950+0000 7fbf71ffb700 10 client.?.objecter resend_mon_ops 2024-04-24T17:28:56.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:56.950+0000 7fbf71ffb700 1 -- 172.21.15.32:0/3175493670 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1844828461 0 0) 0x7fbf7419a990 con 0x7fbf74199ea0 2024-04-24T17:28:56.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:56.950+0000 7fbf71ffb700 1 -- 172.21.15.32:0/3175493670 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf60005730 con 0x7fbf74199ea0 2024-04-24T17:28:56.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:56.950+0000 7fbf71ffb700 1 -- 172.21.15.32:0/3175493670 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 702791276 0 0) 0x7fbf60005730 con 0x7fbf74199ea0 2024-04-24T17:28:56.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:56.950+0000 7fbf71ffb700 1 -- 172.21.15.32:0/3175493670 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbf7419a990 con 0x7fbf74199ea0 2024-04-24T17:28:56.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:56.950+0000 7fbf71ffb700 1 -- 172.21.15.32:0/3175493670 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fbf640047d0 con 0x7fbf74199ea0 2024-04-24T17:28:56.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:56.950+0000 7fbf71ffb700 1 -- 172.21.15.32:0/3175493670 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3632141107 0 0) 0x7fbf7419a990 con 0x7fbf74199ea0 2024-04-24T17:28:56.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:56.950+0000 7fbf71ffb700 1 -- 172.21.15.32:0/3175493670 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf741a3a10 con 0x7fbf74199ea0 2024-04-24T17:28:56.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:56.950+0000 7fbf7ad07700 1 -- 172.21.15.32:0/3175493670 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbf741a2a00 con 0x7fbf74199ea0 2024-04-24T17:28:56.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:56.950+0000 7fbf7ad07700 10 client.14180.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-24T17:28:56.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:56.950+0000 7fbf7ad07700 1 -- 172.21.15.32:0/3175493670 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbf741a2ec0 con 0x7fbf74199ea0 2024-04-24T17:28:56.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:56.950+0000 7fbf7ad07700 1 librados: init done 2024-04-24T17:28:56.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:56.950+0000 7fbf71ffb700 1 -- 172.21.15.32:0/3175493670 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(25 keys) ==== 1042+0+0 (unknown 2768897130 0 0) 0x7fbf64001fe0 con 0x7fbf74199ea0 2024-04-24T17:28:56.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:56.950+0000 7fbf71ffb700 1 -- 172.21.15.32:0/3175493670 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fbf64004f20 con 0x7fbf74199ea0 2024-04-24T17:28:56.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:56.950+0000 7fbf71ffb700 1 -- 172.21.15.32:0/3175493670 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 13) ==== 46579+0+0 (unknown 583022566 0 0) 0x7fbf640051a0 con 0x7fbf74199ea0 2024-04-24T17:28:56.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:56.950+0000 7fbf7ad07700 1 -- 172.21.15.32:0/3175493670 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf54005120 con 0x7fbf74199ea0 2024-04-24T17:28:56.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:56.950+0000 7fbf71ffb700 1 -- 172.21.15.32:0/3175493670 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 1575289653 0 0) 0x7fbf640483a0 con 0x7fbf74199ea0 2024-04-24T17:28:56.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:56.950+0000 7fbf71ffb700 10 client.14180.objecter ms_dispatch 0x7fbf74001550 osd_map(3..3 src has 1..3) 2024-04-24T17:28:56.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:56.950+0000 7fbf71ffb700 3 client.14180.objecter handle_osd_map got epochs [3,3] > 0 2024-04-24T17:28:56.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:56.950+0000 7fbf71ffb700 3 client.14180.objecter handle_osd_map decoding full epoch 3 2024-04-24T17:28:56.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:56.950+0000 7fbf71ffb700 20 client.14180.objecter dump_active .. 0 homeless 2024-04-24T17:28:56.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:56.950+0000 7fbf71ffb700 10 client.14180.objecter ms_handle_connect 0x7fbf6003c2c0 2024-04-24T17:28:56.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:56.958+0000 7fbf71ffb700 1 -- 172.21.15.32:0/3175493670 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188728 (unknown 1092875540 0 2656817166) 0x7fbf64017960 con 0x7fbf74199ea0 2024-04-24T17:28:57.001 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:56 smithi032 bash[21457]: cluster 2024-04-24T17:28:55.392000+0000 mgr.y (mgr.14152) 32 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:28:57.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:56 smithi032 bash[21457]: cluster 2024-04-24T17:28:55.392000+0000 mgr.y (mgr.14152) 32 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:28:57.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:57.178+0000 7fbf7ad07700 1 -- 172.21.15.32:0/3175493670 --> v1:172.21.15.32:6800/1847389373 -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbf54000b90 con 0x7fbf6003c2c0 2024-04-24T17:28:57.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:57.178+0000 7fbf71ffb700 1 -- 172.21.15.32:0/3175493670 <== mgr.14152 v1:172.21.15.32:6800/1847389373 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+159 (unknown 0 0 816037817) 0x7fbf54000b90 con 0x7fbf6003c2c0 2024-04-24T17:28:57.180 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-24T17:28:57.180 INFO:teuthology.orchestra.run.smithi032.stdout:[{"addr": "172.21.15.32", "hostname": "smithi032", "labels": [], "status": ""}, {"addr": "172.21.15.82", "hostname": "smithi082", "labels": [], "status": ""}] 2024-04-24T17:28:57.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:57.182+0000 7fbf7ad07700 10 librados: watch_flush enter 2024-04-24T17:28:57.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:57.182+0000 7fbf7ad07700 10 librados: watch_flush exit 2024-04-24T17:28:57.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:57.182+0000 7fbf7ad07700 20 client.14180.objecter shutdown clearing up homeless session... 2024-04-24T17:28:57.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:57.182+0000 7fbf7ad07700 10 client.14180.objecter successfully canceled tick 2024-04-24T17:28:57.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:57.182+0000 7fbf7ad07700 1 -- 172.21.15.32:0/3175493670 >> v1:172.21.15.32:6800/1847389373 conn(0x7fbf6003c2c0 legacy=0x7fbf6003e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:28:57.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:57.182+0000 7fbf7ad07700 1 -- 172.21.15.32:0/3175493670 >> v1:172.21.15.32:6789/0 conn(0x7fbf74199ea0 legacy=0x7fbf7419a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:28:57.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:57.182+0000 7fbf7ad07700 1 -- 172.21.15.32:0/3175493670 shutdown_connections 2024-04-24T17:28:57.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:57.182+0000 7fbf7ad07700 1 -- 172.21.15.32:0/3175493670 >> 172.21.15.32:0/3175493670 conn(0x7fbf740fd7c0 msgr2=0x7fbf740fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:28:57.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:57.182+0000 7fbf7ad07700 1 -- 172.21.15.32:0/3175493670 shutdown_connections 2024-04-24T17:28:57.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:57.182+0000 7fbf7ad07700 1 -- 172.21.15.32:0/3175493670 wait complete. 2024-04-24T17:28:57.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:28:57.182+0000 7fbf7ad07700 1 librados: shutdown 2024-04-24T17:28:58.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:57 smithi032 bash[21457]: audit 2024-04-24T17:28:56.830405+0000 mon.a (mon.0) 121 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:28:58.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:57 smithi032 bash[21457]: audit 2024-04-24T17:28:56.830405+0000 mon.a (mon.0) 121 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:28:58.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:57 smithi032 bash[21457]: audit 2024-04-24T17:28:56.831624+0000 mon.a (mon.0) 122 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi082", "name": "osd_memory_target"}]: dispatch 2024-04-24T17:28:58.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:57 smithi032 bash[21457]: audit 2024-04-24T17:28:56.831624+0000 mon.a (mon.0) 122 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi082", "name": "osd_memory_target"}]: dispatch 2024-04-24T17:28:58.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:57 smithi032 bash[21457]: audit 2024-04-24T17:28:57.180678+0000 mgr.y (mgr.14152) 33 : audit [DBG] from='client.14180 v1:172.21.15.32:0/3175493670' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T17:28:58.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:57 smithi032 bash[21457]: audit 2024-04-24T17:28:57.180678+0000 mgr.y (mgr.14152) 33 : audit [DBG] from='client.14180 v1:172.21.15.32:0/3175493670' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T17:28:58.099 INFO:tasks.cephadm:Setting crush tunables to jewel 2024-04-24T17:28:58.100 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a7981b84-025f-11ef-bc93-c7b262605968 -- ceph osd crush tunables jewel 2024-04-24T17:28:59.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:58 smithi032 bash[21457]: cluster 2024-04-24T17:28:57.392705+0000 mgr.y (mgr.14152) 34 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:28:59.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:58 smithi032 bash[21457]: cluster 2024-04-24T17:28:57.392705+0000 mgr.y (mgr.14152) 34 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:28:59.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:58 smithi032 bash[21457]: audit 2024-04-24T17:28:58.109403+0000 mon.a (mon.0) 123 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:28:59.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:58 smithi032 bash[21457]: audit 2024-04-24T17:28:58.109403+0000 mon.a (mon.0) 123 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:28:59.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:58 smithi032 bash[21457]: audit 2024-04-24T17:28:58.111049+0000 mon.a (mon.0) 124 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T17:28:59.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:58 smithi032 bash[21457]: audit 2024-04-24T17:28:58.111049+0000 mon.a (mon.0) 124 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T17:28:59.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:58 smithi032 bash[21457]: cephadm 2024-04-24T17:28:58.113585+0000 mgr.y (mgr.14152) 35 : cephadm [INF] Updating smithi082:/etc/ceph/ceph.conf 2024-04-24T17:28:59.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:58 smithi032 bash[21457]: cephadm 2024-04-24T17:28:58.113585+0000 mgr.y (mgr.14152) 35 : cephadm [INF] Updating smithi082:/etc/ceph/ceph.conf 2024-04-24T17:28:59.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:58 smithi032 bash[21457]: cephadm 2024-04-24T17:28:58.328659+0000 mgr.y (mgr.14152) 36 : cephadm [INF] Updating smithi082:/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T17:28:59.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:58 smithi032 bash[21457]: cephadm 2024-04-24T17:28:58.328659+0000 mgr.y (mgr.14152) 36 : cephadm [INF] Updating smithi082:/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T17:28:59.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:58 smithi032 bash[21457]: audit 2024-04-24T17:28:58.829024+0000 mon.a (mon.0) 125 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:28:59.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:58 smithi032 bash[21457]: audit 2024-04-24T17:28:58.829024+0000 mon.a (mon.0) 125 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:00.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:59 smithi032 bash[21457]: cephadm 2024-04-24T17:28:58.493807+0000 mgr.y (mgr.14152) 37 : cephadm [INF] Updating smithi082:/etc/ceph/ceph.client.admin.keyring 2024-04-24T17:29:00.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:59 smithi032 bash[21457]: cephadm 2024-04-24T17:28:58.493807+0000 mgr.y (mgr.14152) 37 : cephadm [INF] Updating smithi082:/etc/ceph/ceph.client.admin.keyring 2024-04-24T17:29:00.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:59 smithi032 bash[21457]: cephadm 2024-04-24T17:28:58.658745+0000 mgr.y (mgr.14152) 38 : cephadm [INF] Updating smithi082:/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T17:29:00.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:59 smithi032 bash[21457]: cephadm 2024-04-24T17:28:58.658745+0000 mgr.y (mgr.14152) 38 : cephadm [INF] Updating smithi082:/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T17:29:00.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:59 smithi032 bash[21457]: cluster 2024-04-24T17:28:58.830203+0000 mgr.y (mgr.14152) 39 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:00.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:59 smithi032 bash[21457]: cluster 2024-04-24T17:28:58.830203+0000 mgr.y (mgr.14152) 39 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:00.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:59 smithi032 bash[21457]: cluster 2024-04-24T17:28:58.831393+0000 mgr.y (mgr.14152) 40 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:00.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:59 smithi032 bash[21457]: cluster 2024-04-24T17:28:58.831393+0000 mgr.y (mgr.14152) 40 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:00.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:59 smithi032 bash[21457]: audit 2024-04-24T17:28:58.836349+0000 mon.a (mon.0) 126 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:00.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:28:59 smithi032 bash[21457]: audit 2024-04-24T17:28:58.836349+0000 mon.a (mon.0) 126 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:02.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:01 smithi032 bash[21457]: cluster 2024-04-24T17:29:00.831955+0000 mgr.y (mgr.14152) 41 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:02.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:01 smithi032 bash[21457]: cluster 2024-04-24T17:29:00.831955+0000 mgr.y (mgr.14152) 41 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:02.818 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T17:29:03.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:03.870+0000 7f2bcf2d6700 1 -- 172.21.15.32:0/1057934234 >> v1:172.21.15.32:6789/0 conn(0x7f2bc810f430 legacy=0x7f2bc810f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:29:03.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:03.870+0000 7f2bcf2d6700 1 -- 172.21.15.32:0/1057934234 shutdown_connections 2024-04-24T17:29:03.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:03.870+0000 7f2bcf2d6700 1 -- 172.21.15.32:0/1057934234 >> 172.21.15.32:0/1057934234 conn(0x7f2bc8078bc0 msgr2=0x7f2bc807afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:29:03.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:03.870+0000 7f2bcf2d6700 1 -- 172.21.15.32:0/1057934234 shutdown_connections 2024-04-24T17:29:03.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:03.870+0000 7f2bcf2d6700 1 -- 172.21.15.32:0/1057934234 wait complete. 2024-04-24T17:29:03.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:03.870+0000 7f2bcf2d6700 1 librados: starting msgr at 2024-04-24T17:29:03.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:03.870+0000 7f2bcf2d6700 1 librados: starting objecter 2024-04-24T17:29:03.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:03.870+0000 7f2bcf2d6700 1 Processor -- start 2024-04-24T17:29:03.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:03.870+0000 7f2bcf2d6700 1 -- start start 2024-04-24T17:29:03.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:03.870+0000 7f2bcf2d6700 1 librados: setting wanted keys 2024-04-24T17:29:03.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:03.870+0000 7f2bcf2d6700 1 librados: calling monclient init 2024-04-24T17:29:03.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:03.870+0000 7f2bcf2d6700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2bc819ea00 con 0x7f2bc810f430 2024-04-24T17:29:03.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:03.870+0000 7f2bcd072700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f2bc810f430 0x7f2bc819e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51080/0 (socket says 172.21.15.32:51080) 2024-04-24T17:29:03.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:03.870+0000 7f2bcd072700 1 -- 172.21.15.32:0/3025043927 learned_addr learned my addr 172.21.15.32:0/3025043927 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:29:03.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:03.870+0000 7f2bbe7fc700 10 client.?.objecter ms_handle_connect 0x7f2bc810f430 2024-04-24T17:29:03.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:03.870+0000 7f2bbe7fc700 10 client.?.objecter resend_mon_ops 2024-04-24T17:29:03.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:03.870+0000 7f2bbe7fc700 1 -- 172.21.15.32:0/3025043927 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1106711998 0 0) 0x7f2bc819ea00 con 0x7f2bc810f430 2024-04-24T17:29:03.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:03.870+0000 7f2bbe7fc700 1 -- 172.21.15.32:0/3025043927 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2bb40056e0 con 0x7f2bc810f430 2024-04-24T17:29:03.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:03.870+0000 7f2bbe7fc700 1 -- 172.21.15.32:0/3025043927 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 727846669 0 0) 0x7f2bb40056e0 con 0x7f2bc810f430 2024-04-24T17:29:03.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:03.870+0000 7f2bbe7fc700 1 -- 172.21.15.32:0/3025043927 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2bc819ea00 con 0x7f2bc810f430 2024-04-24T17:29:03.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:03.870+0000 7f2bbe7fc700 1 -- 172.21.15.32:0/3025043927 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f2bc4004800 con 0x7f2bc810f430 2024-04-24T17:29:03.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:03.870+0000 7f2bbe7fc700 1 -- 172.21.15.32:0/3025043927 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 317691265 0 0) 0x7f2bc819ea00 con 0x7f2bc810f430 2024-04-24T17:29:03.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:03.870+0000 7f2bbe7fc700 1 -- 172.21.15.32:0/3025043927 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bc819ebd0 con 0x7f2bc810f430 2024-04-24T17:29:03.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:03.870+0000 7f2bcf2d6700 1 -- 172.21.15.32:0/3025043927 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2bc819eec0 con 0x7f2bc810f430 2024-04-24T17:29:03.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:03.870+0000 7f2bcf2d6700 10 client.14182.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-24T17:29:03.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:03.870+0000 7f2bcf2d6700 1 -- 172.21.15.32:0/3025043927 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2bc81a7080 con 0x7f2bc810f430 2024-04-24T17:29:03.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:03.870+0000 7f2bcf2d6700 1 librados: init done 2024-04-24T17:29:03.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:03.870+0000 7f2bbe7fc700 1 -- 172.21.15.32:0/3025043927 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(25 keys) ==== 1042+0+0 (unknown 2768897130 0 0) 0x7f2bc4001a80 con 0x7f2bc810f430 2024-04-24T17:29:03.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:03.870+0000 7f2bbe7fc700 1 -- 172.21.15.32:0/3025043927 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f2bc4004d00 con 0x7f2bc810f430 2024-04-24T17:29:03.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:03.874+0000 7f2bbe7fc700 1 -- 172.21.15.32:0/3025043927 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 13) ==== 46579+0+0 (unknown 583022566 0 0) 0x7f2bc4004ec0 con 0x7f2bc810f430 2024-04-24T17:29:03.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:03.874+0000 7f2bcf2d6700 1 -- 172.21.15.32:0/3025043927 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bac005120 con 0x7f2bc810f430 2024-04-24T17:29:03.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:03.874+0000 7f2bbe7fc700 1 -- 172.21.15.32:0/3025043927 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 1575289653 0 0) 0x7f2bc4048060 con 0x7f2bc810f430 2024-04-24T17:29:03.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:03.874+0000 7f2bbe7fc700 10 client.14182.objecter ms_dispatch 0x7f2bc8001550 osd_map(3..3 src has 1..3) 2024-04-24T17:29:03.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:03.874+0000 7f2bbe7fc700 3 client.14182.objecter handle_osd_map got epochs [3,3] > 0 2024-04-24T17:29:03.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:03.874+0000 7f2bbe7fc700 3 client.14182.objecter handle_osd_map decoding full epoch 3 2024-04-24T17:29:03.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:03.874+0000 7f2bbe7fc700 20 client.14182.objecter dump_active .. 0 homeless 2024-04-24T17:29:03.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:03.874+0000 7f2bbe7fc700 10 client.14182.objecter ms_handle_connect 0x7f2bb403c2f0 2024-04-24T17:29:03.879 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:03.878+0000 7f2bbe7fc700 1 -- 172.21.15.32:0/3025043927 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188728 (unknown 1092875540 0 2656817166) 0x7f2bc810ffb0 con 0x7f2bc810f430 2024-04-24T17:29:04.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:03 smithi032 bash[21457]: cluster 2024-04-24T17:29:02.832768+0000 mgr.y (mgr.14152) 42 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:04.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:03 smithi032 bash[21457]: cluster 2024-04-24T17:29:02.832768+0000 mgr.y (mgr.14152) 42 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:04.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:04.106+0000 7f2bcf2d6700 1 -- 172.21.15.32:0/3025043927 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "jewel"} v 0) -- 0x7f2bac005410 con 0x7f2bc810f430 2024-04-24T17:29:04.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:04.846+0000 7f2bbe7fc700 1 -- 172.21.15.32:0/3025043927 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "jewel"}]=0 adjusted tunables profile to jewel v4) ==== 120+0+0 (unknown 2099115640 0 0) 0x7f2bc4017610 con 0x7f2bc810f430 2024-04-24T17:29:04.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:04.850+0000 7f2bcf2d6700 10 librados: watch_flush enter 2024-04-24T17:29:04.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:04.850+0000 7f2bcf2d6700 10 librados: watch_flush exit 2024-04-24T17:29:04.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:04.850+0000 7f2bcf2d6700 20 client.14182.objecter shutdown clearing up homeless session... 2024-04-24T17:29:04.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:04.850+0000 7f2bcf2d6700 10 client.14182.objecter successfully canceled tick 2024-04-24T17:29:04.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:04.850+0000 7f2bcf2d6700 1 -- 172.21.15.32:0/3025043927 >> v1:172.21.15.32:6800/1847389373 conn(0x7f2bb403c2f0 legacy=0x7f2bb403e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:29:04.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:04.850+0000 7f2bcf2d6700 1 -- 172.21.15.32:0/3025043927 >> v1:172.21.15.32:6789/0 conn(0x7f2bc810f430 legacy=0x7f2bc819e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:29:04.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:04.850+0000 7f2bcf2d6700 1 -- 172.21.15.32:0/3025043927 shutdown_connections 2024-04-24T17:29:04.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:04.850+0000 7f2bcf2d6700 1 -- 172.21.15.32:0/3025043927 >> 172.21.15.32:0/3025043927 conn(0x7f2bc8078bc0 msgr2=0x7f2bc81108a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:29:04.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:04.850+0000 7f2bcf2d6700 1 -- 172.21.15.32:0/3025043927 shutdown_connections 2024-04-24T17:29:04.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:04.850+0000 7f2bcf2d6700 1 -- 172.21.15.32:0/3025043927 wait complete. 2024-04-24T17:29:04.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:04.850+0000 7f2bcf2d6700 1 librados: shutdown 2024-04-24T17:29:04.853 INFO:teuthology.orchestra.run.smithi032.stderr:adjusted tunables profile to jewel 2024-04-24T17:29:05.140 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:04 smithi032 bash[21457]: audit 2024-04-24T17:29:04.107358+0000 mon.a (mon.0) 127 : audit [INF] from='client.? v1:172.21.15.32:0/3025043927' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "jewel"}]: dispatch 2024-04-24T17:29:05.140 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:04 smithi032 bash[21457]: audit 2024-04-24T17:29:04.107358+0000 mon.a (mon.0) 127 : audit [INF] from='client.? v1:172.21.15.32:0/3025043927' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "jewel"}]: dispatch 2024-04-24T17:29:05.604 INFO:tasks.cephadm:Adding mon.a on smithi032 2024-04-24T17:29:05.604 INFO:tasks.cephadm:Adding mon.c on smithi032 2024-04-24T17:29:05.604 INFO:tasks.cephadm:Adding mon.b on smithi082 2024-04-24T17:29:05.605 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a7981b84-025f-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi032:[v1:172.21.15.32:6789]=a;smithi032:[v1:172.21.15.32:6790]=c;smithi082:[v1:172.21.15.82:6789]=b' 2024-04-24T17:29:06.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:05 smithi032 bash[21457]: cluster 2024-04-24T17:29:04.833787+0000 mgr.y (mgr.14152) 43 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:06.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:05 smithi032 bash[21457]: cluster 2024-04-24T17:29:04.833787+0000 mgr.y (mgr.14152) 43 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:06.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:05 smithi032 bash[21457]: audit 2024-04-24T17:29:04.848534+0000 mon.a (mon.0) 128 : audit [INF] from='client.? v1:172.21.15.32:0/3025043927' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "jewel"}]': finished 2024-04-24T17:29:06.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:05 smithi032 bash[21457]: audit 2024-04-24T17:29:04.848534+0000 mon.a (mon.0) 128 : audit [INF] from='client.? v1:172.21.15.32:0/3025043927' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "jewel"}]': finished 2024-04-24T17:29:06.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:05 smithi032 bash[21457]: cluster 2024-04-24T17:29:04.853268+0000 mon.a (mon.0) 129 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-04-24T17:29:06.002 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:05 smithi032 bash[21457]: cluster 2024-04-24T17:29:04.853268+0000 mon.a (mon.0) 129 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-04-24T17:29:06.811 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T17:29:07.733 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.730+0000 7fe8156d9700 1 -- 172.21.15.82:0/1433578674 >> v1:172.21.15.32:6789/0 conn(0x7fe810102fd0 legacy=0x7fe8101033b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:29:07.733 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.730+0000 7fe8156d9700 1 -- 172.21.15.82:0/1433578674 shutdown_connections 2024-04-24T17:29:07.733 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.730+0000 7fe8156d9700 1 -- 172.21.15.82:0/1433578674 >> 172.21.15.82:0/1433578674 conn(0x7fe8100fe8a0 msgr2=0x7fe810100cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:29:07.733 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.730+0000 7fe8156d9700 1 -- 172.21.15.82:0/1433578674 shutdown_connections 2024-04-24T17:29:07.733 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.730+0000 7fe8156d9700 1 -- 172.21.15.82:0/1433578674 wait complete. 2024-04-24T17:29:07.733 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.730+0000 7fe8156d9700 1 librados: starting msgr at 2024-04-24T17:29:07.733 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.730+0000 7fe8156d9700 1 librados: starting objecter 2024-04-24T17:29:07.734 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.730+0000 7fe8156d9700 1 Processor -- start 2024-04-24T17:29:07.734 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.730+0000 7fe8156d9700 1 -- start start 2024-04-24T17:29:07.734 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.730+0000 7fe8156d9700 1 librados: setting wanted keys 2024-04-24T17:29:07.734 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.730+0000 7fe8156d9700 1 librados: calling monclient init 2024-04-24T17:29:07.734 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.730+0000 7fe8156d9700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe8101a0050 con 0x7fe810102fd0 2024-04-24T17:29:07.735 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.730+0000 7fe80effd700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe810102fd0 0x7fe81019f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.82:54902/0 (socket says 172.21.15.82:54902) 2024-04-24T17:29:07.735 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.730+0000 7fe80effd700 1 -- 172.21.15.82:0/3357327228 learned_addr learned my addr 172.21.15.82:0/3357327228 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-04-24T17:29:07.735 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.730+0000 7fe7f7fff700 10 client.?.objecter ms_handle_connect 0x7fe810102fd0 2024-04-24T17:29:07.735 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.730+0000 7fe7f7fff700 10 client.?.objecter resend_mon_ops 2024-04-24T17:29:07.736 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.730+0000 7fe7f7fff700 1 -- 172.21.15.82:0/3357327228 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2813949730 0 0) 0x7fe8101a0050 con 0x7fe810102fd0 2024-04-24T17:29:07.736 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.730+0000 7fe7f7fff700 1 -- 172.21.15.82:0/3357327228 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe7f80056e0 con 0x7fe810102fd0 2024-04-24T17:29:07.736 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.730+0000 7fe7f7fff700 1 -- 172.21.15.82:0/3357327228 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2594089887 0 0) 0x7fe7f80056e0 con 0x7fe810102fd0 2024-04-24T17:29:07.736 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.730+0000 7fe7f7fff700 1 -- 172.21.15.82:0/3357327228 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe8101a0050 con 0x7fe810102fd0 2024-04-24T17:29:07.736 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.730+0000 7fe7f7fff700 1 -- 172.21.15.82:0/3357327228 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fe8000049b0 con 0x7fe810102fd0 2024-04-24T17:29:07.736 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.730+0000 7fe7f7fff700 1 -- 172.21.15.82:0/3357327228 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1921809185 0 0) 0x7fe8101a0050 con 0x7fe810102fd0 2024-04-24T17:29:07.736 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.730+0000 7fe7f7fff700 1 -- 172.21.15.82:0/3357327228 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8101a0220 con 0x7fe810102fd0 2024-04-24T17:29:07.736 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.730+0000 7fe8156d9700 1 -- 172.21.15.82:0/3357327228 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe8101a0510 con 0x7fe810102fd0 2024-04-24T17:29:07.736 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.730+0000 7fe8156d9700 10 client.14184.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-24T17:29:07.736 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.730+0000 7fe8156d9700 1 -- 172.21.15.82:0/3357327228 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe8101a2a80 con 0x7fe810102fd0 2024-04-24T17:29:07.736 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.730+0000 7fe8156d9700 1 librados: init done 2024-04-24T17:29:07.736 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.730+0000 7fe7f7fff700 1 -- 172.21.15.82:0/3357327228 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(25 keys) ==== 1042+0+0 (unknown 2768897130 0 0) 0x7fe800002090 con 0x7fe810102fd0 2024-04-24T17:29:07.736 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.730+0000 7fe7f7fff700 1 -- 172.21.15.82:0/3357327228 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7fe800004fe0 con 0x7fe810102fd0 2024-04-24T17:29:07.736 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.734+0000 7fe8156d9700 1 -- 172.21.15.82:0/3357327228 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7fc005120 con 0x7fe810102fd0 2024-04-24T17:29:07.737 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.734+0000 7fe7f7fff700 1 -- 172.21.15.82:0/3357327228 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 13) ==== 46579+0+0 (unknown 583022566 0 0) 0x7fe800003ba0 con 0x7fe810102fd0 2024-04-24T17:29:07.737 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.734+0000 7fe7f7fff700 1 -- 172.21.15.82:0/3357327228 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(4..4 src has 1..4) ==== 1069+0+0 (unknown 1280957937 0 0) 0x7fe800048230 con 0x7fe810102fd0 2024-04-24T17:29:07.737 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.734+0000 7fe7f7fff700 10 client.14184.objecter ms_dispatch 0x7fe810001550 osd_map(4..4 src has 1..4) 2024-04-24T17:29:07.737 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.734+0000 7fe7f7fff700 3 client.14184.objecter handle_osd_map got epochs [4,4] > 0 2024-04-24T17:29:07.737 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.734+0000 7fe7f7fff700 3 client.14184.objecter handle_osd_map decoding full epoch 4 2024-04-24T17:29:07.737 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.734+0000 7fe7f7fff700 20 client.14184.objecter dump_active .. 0 homeless 2024-04-24T17:29:07.738 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.734+0000 7fe7f7fff700 10 client.14184.objecter ms_handle_connect 0x7fe7f803c340 2024-04-24T17:29:07.751 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.746+0000 7fe7f7fff700 1 -- 172.21.15.82:0/3357327228 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188728 (unknown 1092875540 0 2656817166) 0x7fe800016940 con 0x7fe810102fd0 2024-04-24T17:29:07.962 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.958+0000 7fe8156d9700 1 -- 172.21.15.82:0/3357327228 --> v1:172.21.15.32:6800/1847389373 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi032:[v1:172.21.15.32:6789]=a;smithi032:[v1:172.21.15.32:6790]=c;smithi082:[v1:172.21.15.82:6789]=b", "target": ["mon-mgr", ""]}) -- 0x7fe7fc000c60 con 0x7fe7f803c340 2024-04-24T17:29:07.977 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.978+0000 7fe7f7fff700 1 -- 172.21.15.82:0/3357327228 <== mgr.14152 v1:172.21.15.32:6800/1847389373 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (unknown 0 0 3265049985) 0x7fe7fc000c60 con 0x7fe7f803c340 2024-04-24T17:29:07.978 INFO:teuthology.orchestra.run.smithi082.stdout:Scheduled mon update... 2024-04-24T17:29:07.980 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.978+0000 7fe8156d9700 10 librados: watch_flush enter 2024-04-24T17:29:07.980 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.978+0000 7fe8156d9700 10 librados: watch_flush exit 2024-04-24T17:29:07.980 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.978+0000 7fe8156d9700 20 client.14184.objecter shutdown clearing up homeless session... 2024-04-24T17:29:07.980 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.978+0000 7fe8156d9700 10 client.14184.objecter successfully canceled tick 2024-04-24T17:29:07.980 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.978+0000 7fe8156d9700 1 -- 172.21.15.82:0/3357327228 >> v1:172.21.15.32:6800/1847389373 conn(0x7fe7f803c340 legacy=0x7fe7f803e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:29:07.980 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.978+0000 7fe8156d9700 1 -- 172.21.15.82:0/3357327228 >> v1:172.21.15.32:6789/0 conn(0x7fe810102fd0 legacy=0x7fe81019f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:29:07.980 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.978+0000 7fe8156d9700 1 -- 172.21.15.82:0/3357327228 shutdown_connections 2024-04-24T17:29:07.980 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.978+0000 7fe8156d9700 1 -- 172.21.15.82:0/3357327228 >> 172.21.15.82:0/3357327228 conn(0x7fe8100fe8a0 msgr2=0x7fe810109f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:29:07.980 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.978+0000 7fe8156d9700 1 -- 172.21.15.82:0/3357327228 shutdown_connections 2024-04-24T17:29:07.980 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.978+0000 7fe8156d9700 1 -- 172.21.15.82:0/3357327228 wait complete. 2024-04-24T17:29:07.980 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:07.978+0000 7fe8156d9700 1 librados: shutdown 2024-04-24T17:29:08.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:07 smithi032 bash[21457]: cluster 2024-04-24T17:29:06.834835+0000 mgr.y (mgr.14152) 44 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:08.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:07 smithi032 bash[21457]: cluster 2024-04-24T17:29:06.834835+0000 mgr.y (mgr.14152) 44 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:08.539 DEBUG:teuthology.orchestra.run.smithi032:mon.c> sudo journalctl -f -n 0 -u ceph-a7981b84-025f-11ef-bc93-c7b262605968@mon.c.service 2024-04-24T17:29:08.542 DEBUG:teuthology.orchestra.run.smithi082:mon.b> sudo journalctl -f -n 0 -u ceph-a7981b84-025f-11ef-bc93-c7b262605968@mon.b.service 2024-04-24T17:29:08.544 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-24T17:29:08.545 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a7981b84-025f-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-24T17:29:09.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:08 smithi032 bash[21457]: audit 2024-04-24T17:29:07.966095+0000 mgr.y (mgr.14152) 45 : audit [DBG] from='client.14184 v1:172.21.15.82:0/3357327228' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi032:[v1:172.21.15.32:6789]=a;smithi032:[v1:172.21.15.32:6790]=c;smithi082:[v1:172.21.15.82:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T17:29:09.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:08 smithi032 bash[21457]: audit 2024-04-24T17:29:07.966095+0000 mgr.y (mgr.14152) 45 : audit [DBG] from='client.14184 v1:172.21.15.82:0/3357327228' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi032:[v1:172.21.15.32:6789]=a;smithi032:[v1:172.21.15.32:6790]=c;smithi082:[v1:172.21.15.82:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T17:29:09.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:08 smithi032 bash[21457]: cephadm 2024-04-24T17:29:07.972279+0000 mgr.y (mgr.14152) 46 : cephadm [INF] Saving service mon spec with placement smithi032:[v1:172.21.15.32:6789]=a;smithi032:[v1:172.21.15.32:6790]=c;smithi082:[v1:172.21.15.82:6789]=b;count:3 2024-04-24T17:29:09.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:08 smithi032 bash[21457]: cephadm 2024-04-24T17:29:07.972279+0000 mgr.y (mgr.14152) 46 : cephadm [INF] Saving service mon spec with placement smithi032:[v1:172.21.15.32:6789]=a;smithi032:[v1:172.21.15.32:6790]=c;smithi082:[v1:172.21.15.82:6789]=b;count:3 2024-04-24T17:29:09.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:08 smithi032 bash[21457]: audit 2024-04-24T17:29:07.977751+0000 mon.a (mon.0) 130 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:09.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:08 smithi032 bash[21457]: audit 2024-04-24T17:29:07.977751+0000 mon.a (mon.0) 130 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:09.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:08 smithi032 bash[21457]: audit 2024-04-24T17:29:07.991758+0000 mon.a (mon.0) 131 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T17:29:09.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:08 smithi032 bash[21457]: audit 2024-04-24T17:29:07.991758+0000 mon.a (mon.0) 131 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T17:29:09.781 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T17:29:10.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:09 smithi032 bash[21457]: cluster 2024-04-24T17:29:08.835860+0000 mgr.y (mgr.14152) 47 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:10.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:09 smithi032 bash[21457]: cluster 2024-04-24T17:29:08.835860+0000 mgr.y (mgr.14152) 47 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:10.790 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:10.786+0000 7f3f01d64700 1 -- 172.21.15.82:0/2267602902 >> v1:172.21.15.32:6789/0 conn(0x7f3efc1051d0 legacy=0x7f3efc1055b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:29:10.790 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:10.786+0000 7f3f01d64700 1 -- 172.21.15.82:0/2267602902 shutdown_connections 2024-04-24T17:29:10.790 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:10.786+0000 7f3f01d64700 1 -- 172.21.15.82:0/2267602902 >> 172.21.15.82:0/2267602902 conn(0x7f3efc100aa0 msgr2=0x7f3efc102ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:29:10.790 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:10.786+0000 7f3f01d64700 1 -- 172.21.15.82:0/2267602902 shutdown_connections 2024-04-24T17:29:10.790 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:10.786+0000 7f3f01d64700 1 -- 172.21.15.82:0/2267602902 wait complete. 2024-04-24T17:29:10.791 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:10.786+0000 7f3f01d64700 1 librados: starting msgr at 2024-04-24T17:29:10.791 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:10.786+0000 7f3f01d64700 1 librados: starting objecter 2024-04-24T17:29:10.791 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:10.786+0000 7f3f01d64700 1 Processor -- start 2024-04-24T17:29:10.791 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:10.786+0000 7f3f01d64700 1 -- start start 2024-04-24T17:29:10.791 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:10.786+0000 7f3f01d64700 1 librados: setting wanted keys 2024-04-24T17:29:10.791 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:10.786+0000 7f3f01d64700 1 librados: calling monclient init 2024-04-24T17:29:10.792 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:10.786+0000 7f3f01d64700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3efc19a470 con 0x7f3efc1051d0 2024-04-24T17:29:10.792 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:10.786+0000 7f3efb7fe700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3efc1051d0 0x7f3efc199d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.82:54912/0 (socket says 172.21.15.82:54912) 2024-04-24T17:29:10.792 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:10.786+0000 7f3efb7fe700 1 -- 172.21.15.82:0/1969215094 learned_addr learned my addr 172.21.15.82:0/1969215094 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-04-24T17:29:10.792 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:10.786+0000 7f3ef97fa700 10 client.?.objecter ms_handle_connect 0x7f3efc1051d0 2024-04-24T17:29:10.792 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:10.786+0000 7f3ef97fa700 10 client.?.objecter resend_mon_ops 2024-04-24T17:29:10.792 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:10.790+0000 7f3ef97fa700 1 -- 172.21.15.82:0/1969215094 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3238548545 0 0) 0x7f3efc19a470 con 0x7f3efc1051d0 2024-04-24T17:29:10.792 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:10.790+0000 7f3ef97fa700 1 -- 172.21.15.82:0/1969215094 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ee8005730 con 0x7f3efc1051d0 2024-04-24T17:29:10.792 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:10.790+0000 7f3ef97fa700 1 -- 172.21.15.82:0/1969215094 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2271246314 0 0) 0x7f3ee8005730 con 0x7f3efc1051d0 2024-04-24T17:29:10.792 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:10.790+0000 7f3ef97fa700 1 -- 172.21.15.82:0/1969215094 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3efc19a470 con 0x7f3efc1051d0 2024-04-24T17:29:10.792 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:10.790+0000 7f3ef97fa700 1 -- 172.21.15.82:0/1969215094 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f3ee4003790 con 0x7f3efc1051d0 2024-04-24T17:29:10.793 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:10.790+0000 7f3ef97fa700 1 -- 172.21.15.82:0/1969215094 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3699860047 0 0) 0x7f3efc19a470 con 0x7f3efc1051d0 2024-04-24T17:29:10.793 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:10.790+0000 7f3ef97fa700 1 -- 172.21.15.82:0/1969215094 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3efc19a640 con 0x7f3efc1051d0 2024-04-24T17:29:10.794 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:10.790+0000 7f3f01d64700 1 -- 172.21.15.82:0/1969215094 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3efc19a930 con 0x7f3efc1051d0 2024-04-24T17:29:10.794 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:10.790+0000 7f3f01d64700 10 client.14186.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-24T17:29:10.794 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:10.790+0000 7f3f01d64700 1 -- 172.21.15.82:0/1969215094 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3efc1a2af0 con 0x7f3efc1051d0 2024-04-24T17:29:10.794 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:10.790+0000 7f3f01d64700 1 librados: init done 2024-04-24T17:29:10.794 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:10.790+0000 7f3ef97fa700 1 -- 172.21.15.82:0/1969215094 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(25 keys) ==== 1042+0+0 (unknown 2768897130 0 0) 0x7f3ee4001230 con 0x7f3efc1051d0 2024-04-24T17:29:10.794 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:10.790+0000 7f3ef97fa700 1 -- 172.21.15.82:0/1969215094 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f3ee4004b50 con 0x7f3efc1051d0 2024-04-24T17:29:10.794 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:10.790+0000 7f3f01d64700 1 -- 172.21.15.82:0/1969215094 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3edc005120 con 0x7f3efc1051d0 2024-04-24T17:29:10.795 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:10.790+0000 7f3ef97fa700 1 -- 172.21.15.82:0/1969215094 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 13) ==== 46579+0+0 (unknown 583022566 0 0) 0x7f3ee4003dc0 con 0x7f3efc1051d0 2024-04-24T17:29:10.795 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:10.790+0000 7f3ef97fa700 1 -- 172.21.15.82:0/1969215094 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(4..4 src has 1..4) ==== 1069+0+0 (unknown 1280957937 0 0) 0x7f3ee4047df0 con 0x7f3efc1051d0 2024-04-24T17:29:10.795 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:10.790+0000 7f3ef97fa700 10 client.14186.objecter ms_dispatch 0x7f3efc001550 osd_map(4..4 src has 1..4) 2024-04-24T17:29:10.795 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:10.790+0000 7f3ef97fa700 3 client.14186.objecter handle_osd_map got epochs [4,4] > 0 2024-04-24T17:29:10.795 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:10.790+0000 7f3ef97fa700 3 client.14186.objecter handle_osd_map decoding full epoch 4 2024-04-24T17:29:10.795 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:10.790+0000 7f3ef97fa700 20 client.14186.objecter dump_active .. 0 homeless 2024-04-24T17:29:10.795 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:10.794+0000 7f3ef97fa700 10 client.14186.objecter ms_handle_connect 0x7f3ee803c3a0 2024-04-24T17:29:10.808 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:10.806+0000 7f3ef97fa700 1 -- 172.21.15.82:0/1969215094 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188728 (unknown 1092875540 0 2656817166) 0x7f3ee4016560 con 0x7f3efc1051d0 2024-04-24T17:29:11.092 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:11.090+0000 7f3f01d64700 1 -- 172.21.15.82:0/1969215094 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f3edc005410 con 0x7f3efc1051d0 2024-04-24T17:29:11.093 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:11.090+0000 7f3ef97fa700 1 -- 172.21.15.82:0/1969215094 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+694 (unknown 2237029548 0 3694938686) 0x7f3ee4015e60 con 0x7f3efc1051d0 2024-04-24T17:29:11.094 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-24T17:29:11.094 INFO:teuthology.orchestra.run.smithi082.stdout:{"epoch":1,"fsid":"a7981b84-025f-11ef-bc93-c7b262605968","modified":"2024-04-24T17:26:59.605652Z","created":"2024-04-24T17:26:59.605652Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-24T17:29:11.096 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:11.094+0000 7f3f01d64700 10 librados: watch_flush enter 2024-04-24T17:29:11.096 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:11.094+0000 7f3f01d64700 10 librados: watch_flush exit 2024-04-24T17:29:11.096 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:11.094+0000 7f3f01d64700 20 client.14186.objecter shutdown clearing up homeless session... 2024-04-24T17:29:11.096 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:11.094+0000 7f3f01d64700 10 client.14186.objecter successfully canceled tick 2024-04-24T17:29:11.096 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:11.094+0000 7f3f01d64700 1 -- 172.21.15.82:0/1969215094 >> v1:172.21.15.32:6800/1847389373 conn(0x7f3ee803c3a0 legacy=0x7f3ee803e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:29:11.097 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:11.094+0000 7f3f01d64700 1 -- 172.21.15.82:0/1969215094 >> v1:172.21.15.32:6789/0 conn(0x7f3efc1051d0 legacy=0x7f3efc199d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:29:11.097 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:11.094+0000 7f3f01d64700 1 -- 172.21.15.82:0/1969215094 shutdown_connections 2024-04-24T17:29:11.097 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:11.094+0000 7f3f01d64700 1 -- 172.21.15.82:0/1969215094 >> 172.21.15.82:0/1969215094 conn(0x7f3efc100aa0 msgr2=0x7f3efc10c350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:29:11.097 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:11.094+0000 7f3f01d64700 1 -- 172.21.15.82:0/1969215094 shutdown_connections 2024-04-24T17:29:11.097 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:11.094+0000 7f3f01d64700 1 -- 172.21.15.82:0/1969215094 wait complete. 2024-04-24T17:29:11.097 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:11.094+0000 7f3f01d64700 1 librados: shutdown 2024-04-24T17:29:11.097 INFO:teuthology.orchestra.run.smithi082.stderr:dumped monmap epoch 1 2024-04-24T17:29:11.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:11 smithi032 bash[21457]: audit 2024-04-24T17:29:11.094590+0000 mon.a (mon.0) 132 : audit [DBG] from='client.? v1:172.21.15.82:0/1969215094' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T17:29:11.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:11 smithi032 bash[21457]: audit 2024-04-24T17:29:11.094590+0000 mon.a (mon.0) 132 : audit [DBG] from='client.? v1:172.21.15.82:0/1969215094' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T17:29:12.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:12 smithi032 bash[21457]: cluster 2024-04-24T17:29:10.836794+0000 mgr.y (mgr.14152) 48 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:12.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:12 smithi032 bash[21457]: cluster 2024-04-24T17:29:10.836794+0000 mgr.y (mgr.14152) 48 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:12.747 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-24T17:29:12.748 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a7981b84-025f-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-24T17:29:13.951 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T17:29:14.247 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:14 smithi032 bash[21457]: cluster 2024-04-24T17:29:12.837807+0000 mgr.y (mgr.14152) 49 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:14.247 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:14 smithi032 bash[21457]: cluster 2024-04-24T17:29:12.837807+0000 mgr.y (mgr.14152) 49 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:14.843 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:14.837+0000 7f46671c5700 1 -- 172.21.15.82:0/1027059891 >> v1:172.21.15.32:6789/0 conn(0x7f4660102fa0 legacy=0x7f4660103380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:29:14.843 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:14.837+0000 7f46671c5700 1 -- 172.21.15.82:0/1027059891 shutdown_connections 2024-04-24T17:29:14.843 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:14.837+0000 7f46671c5700 1 -- 172.21.15.82:0/1027059891 >> 172.21.15.82:0/1027059891 conn(0x7f46600fe850 msgr2=0x7f4660100c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:29:14.843 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:14.837+0000 7f46671c5700 1 -- 172.21.15.82:0/1027059891 shutdown_connections 2024-04-24T17:29:14.843 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:14.837+0000 7f46671c5700 1 -- 172.21.15.82:0/1027059891 wait complete. 2024-04-24T17:29:14.843 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:14.837+0000 7f46671c5700 1 librados: starting msgr at 2024-04-24T17:29:14.844 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:14.837+0000 7f46671c5700 1 librados: starting objecter 2024-04-24T17:29:14.844 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:14.837+0000 7f46671c5700 1 Processor -- start 2024-04-24T17:29:14.844 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:14.837+0000 7f46671c5700 1 -- start start 2024-04-24T17:29:14.844 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:14.837+0000 7f46671c5700 1 librados: setting wanted keys 2024-04-24T17:29:14.844 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:14.837+0000 7f46671c5700 1 librados: calling monclient init 2024-04-24T17:29:14.844 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:14.841+0000 7f46671c5700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f46600775c0 con 0x7f4660075720 2024-04-24T17:29:14.845 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:14.841+0000 7f4664f61700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f4660075720 0x7f4660075b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.82:41352/0 (socket says 172.21.15.82:41352) 2024-04-24T17:29:14.845 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:14.841+0000 7f4664f61700 1 -- 172.21.15.82:0/2880720608 learned_addr learned my addr 172.21.15.82:0/2880720608 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-04-24T17:29:14.845 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:14.841+0000 7f465dffb700 10 client.?.objecter ms_handle_connect 0x7f4660075720 2024-04-24T17:29:14.845 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:14.841+0000 7f465dffb700 10 client.?.objecter resend_mon_ops 2024-04-24T17:29:14.845 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:14.841+0000 7f465dffb700 1 -- 172.21.15.82:0/2880720608 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1168993628 0 0) 0x7f46600775c0 con 0x7f4660075720 2024-04-24T17:29:14.845 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:14.841+0000 7f465dffb700 1 -- 172.21.15.82:0/2880720608 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46480056e0 con 0x7f4660075720 2024-04-24T17:29:14.845 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:14.841+0000 7f465dffb700 1 -- 172.21.15.82:0/2880720608 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3566022508 0 0) 0x7f46480056e0 con 0x7f4660075720 2024-04-24T17:29:14.845 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:14.841+0000 7f465dffb700 1 -- 172.21.15.82:0/2880720608 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f46600775c0 con 0x7f4660075720 2024-04-24T17:29:14.845 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:14.841+0000 7f465dffb700 1 -- 172.21.15.82:0/2880720608 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f46540043e0 con 0x7f4660075720 2024-04-24T17:29:14.846 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:14.841+0000 7f465dffb700 1 -- 172.21.15.82:0/2880720608 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 307131682 0 0) 0x7f46600775c0 con 0x7f4660075720 2024-04-24T17:29:14.846 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:14.841+0000 7f465dffb700 1 -- 172.21.15.82:0/2880720608 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4660076210 con 0x7f4660075720 2024-04-24T17:29:14.846 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:14.841+0000 7f46671c5700 1 -- 172.21.15.82:0/2880720608 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4660076500 con 0x7f4660075720 2024-04-24T17:29:14.847 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:14.841+0000 7f46671c5700 10 client.14188.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-24T17:29:14.847 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:14.841+0000 7f46671c5700 1 -- 172.21.15.82:0/2880720608 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f46601a3010 con 0x7f4660075720 2024-04-24T17:29:14.847 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:14.841+0000 7f46671c5700 1 librados: init done 2024-04-24T17:29:14.847 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:14.841+0000 7f465dffb700 1 -- 172.21.15.82:0/2880720608 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(25 keys) ==== 1042+0+0 (unknown 2768897130 0 0) 0x7f4654001d10 con 0x7f4660075720 2024-04-24T17:29:14.847 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:14.841+0000 7f465dffb700 1 -- 172.21.15.82:0/2880720608 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f4654004b60 con 0x7f4660075720 2024-04-24T17:29:14.847 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:14.841+0000 7f46671c5700 1 -- 172.21.15.82:0/2880720608 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f464c005120 con 0x7f4660075720 2024-04-24T17:29:14.847 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:14.841+0000 7f465dffb700 1 -- 172.21.15.82:0/2880720608 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 13) ==== 46579+0+0 (unknown 583022566 0 0) 0x7f4654003820 con 0x7f4660075720 2024-04-24T17:29:14.848 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:14.845+0000 7f465dffb700 1 -- 172.21.15.82:0/2880720608 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(4..4 src has 1..4) ==== 1069+0+0 (unknown 1280957937 0 0) 0x7f4654047ed0 con 0x7f4660075720 2024-04-24T17:29:14.848 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:14.845+0000 7f465dffb700 10 client.14188.objecter ms_dispatch 0x7f4660001550 osd_map(4..4 src has 1..4) 2024-04-24T17:29:14.848 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:14.845+0000 7f465dffb700 3 client.14188.objecter handle_osd_map got epochs [4,4] > 0 2024-04-24T17:29:14.848 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:14.845+0000 7f465dffb700 3 client.14188.objecter handle_osd_map decoding full epoch 4 2024-04-24T17:29:14.848 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:14.845+0000 7f465dffb700 20 client.14188.objecter dump_active .. 0 homeless 2024-04-24T17:29:14.848 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:14.845+0000 7f465dffb700 10 client.14188.objecter ms_handle_connect 0x7f464803c2e0 2024-04-24T17:29:14.861 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:14.857+0000 7f465dffb700 1 -- 172.21.15.82:0/2880720608 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188728 (unknown 1092875540 0 2656817166) 0x7f4654016640 con 0x7f4660075720 2024-04-24T17:29:15.151 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:15.149+0000 7f46671c5700 1 -- 172.21.15.82:0/2880720608 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f464c005410 con 0x7f4660075720 2024-04-24T17:29:15.154 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:15.149+0000 7f465dffb700 1 -- 172.21.15.82:0/2880720608 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+694 (unknown 2237029548 0 3694938686) 0x7f4654015f40 con 0x7f4660075720 2024-04-24T17:29:15.154 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-24T17:29:15.154 INFO:teuthology.orchestra.run.smithi082.stdout:{"epoch":1,"fsid":"a7981b84-025f-11ef-bc93-c7b262605968","modified":"2024-04-24T17:26:59.605652Z","created":"2024-04-24T17:26:59.605652Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-24T17:29:15.156 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:15.153+0000 7f46671c5700 10 librados: watch_flush enter 2024-04-24T17:29:15.157 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:15.153+0000 7f46671c5700 10 librados: watch_flush exit 2024-04-24T17:29:15.157 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:15.153+0000 7f46671c5700 20 client.14188.objecter shutdown clearing up homeless session... 2024-04-24T17:29:15.157 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:15.153+0000 7f46671c5700 10 client.14188.objecter successfully canceled tick 2024-04-24T17:29:15.157 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:15.153+0000 7f46671c5700 1 -- 172.21.15.82:0/2880720608 >> v1:172.21.15.32:6800/1847389373 conn(0x7f464803c2e0 legacy=0x7f464803e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:29:15.157 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:15.153+0000 7f46671c5700 1 -- 172.21.15.82:0/2880720608 >> v1:172.21.15.32:6789/0 conn(0x7f4660075720 legacy=0x7f4660075b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:29:15.157 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:15.153+0000 7f46671c5700 1 -- 172.21.15.82:0/2880720608 shutdown_connections 2024-04-24T17:29:15.157 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:15.153+0000 7f46671c5700 1 -- 172.21.15.82:0/2880720608 >> 172.21.15.82:0/2880720608 conn(0x7f46600fe850 msgr2=0x7f4660109c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:29:15.157 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:15.153+0000 7f46671c5700 1 -- 172.21.15.82:0/2880720608 shutdown_connections 2024-04-24T17:29:15.157 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:15.153+0000 7f46671c5700 1 -- 172.21.15.82:0/2880720608 wait complete. 2024-04-24T17:29:15.157 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:15.153+0000 7f46671c5700 1 librados: shutdown 2024-04-24T17:29:15.157 INFO:teuthology.orchestra.run.smithi082.stderr:dumped monmap epoch 1 2024-04-24T17:29:15.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:15 smithi032 bash[21457]: audit 2024-04-24T17:29:15.028832+0000 mon.a (mon.0) 133 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:15.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:15 smithi032 bash[21457]: audit 2024-04-24T17:29:15.028832+0000 mon.a (mon.0) 133 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:15.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:15 smithi032 bash[21457]: audit 2024-04-24T17:29:15.030708+0000 mon.a (mon.0) 134 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T17:29:15.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:15 smithi032 bash[21457]: audit 2024-04-24T17:29:15.030708+0000 mon.a (mon.0) 134 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T17:29:15.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:15 smithi032 bash[21457]: audit 2024-04-24T17:29:15.040843+0000 mon.a (mon.0) 135 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:15.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:15 smithi032 bash[21457]: audit 2024-04-24T17:29:15.040843+0000 mon.a (mon.0) 135 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:15.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:15 smithi032 bash[21457]: audit 2024-04-24T17:29:15.045051+0000 mon.a (mon.0) 136 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T17:29:15.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:15 smithi032 bash[21457]: audit 2024-04-24T17:29:15.045051+0000 mon.a (mon.0) 136 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T17:29:15.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:15 smithi032 bash[21457]: audit 2024-04-24T17:29:15.046772+0000 mon.a (mon.0) 137 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:15.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:15 smithi032 bash[21457]: audit 2024-04-24T17:29:15.046772+0000 mon.a (mon.0) 137 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:16.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:16 smithi032 bash[21457]: cluster 2024-04-24T17:29:14.838776+0000 mgr.y (mgr.14152) 50 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:16.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:16 smithi032 bash[21457]: cluster 2024-04-24T17:29:14.838776+0000 mgr.y (mgr.14152) 50 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:16.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:16 smithi032 bash[21457]: cluster 2024-04-24T17:29:15.033644+0000 mgr.y (mgr.14152) 51 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:16.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:16 smithi032 bash[21457]: cluster 2024-04-24T17:29:15.033644+0000 mgr.y (mgr.14152) 51 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:16.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:16 smithi032 bash[21457]: cluster 2024-04-24T17:29:15.034964+0000 mgr.y (mgr.14152) 52 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:16.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:16 smithi032 bash[21457]: cluster 2024-04-24T17:29:15.034964+0000 mgr.y (mgr.14152) 52 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:16.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:16 smithi032 bash[21457]: cephadm 2024-04-24T17:29:15.048701+0000 mgr.y (mgr.14152) 53 : cephadm [INF] Deploying daemon mon.b on smithi082 2024-04-24T17:29:16.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:16 smithi032 bash[21457]: cephadm 2024-04-24T17:29:15.048701+0000 mgr.y (mgr.14152) 53 : cephadm [INF] Deploying daemon mon.b on smithi082 2024-04-24T17:29:16.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:16 smithi032 bash[21457]: audit 2024-04-24T17:29:15.155067+0000 mon.a (mon.0) 138 : audit [DBG] from='client.? v1:172.21.15.82:0/2880720608' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T17:29:16.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:16 smithi032 bash[21457]: audit 2024-04-24T17:29:15.155067+0000 mon.a (mon.0) 138 : audit [DBG] from='client.? v1:172.21.15.82:0/2880720608' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T17:29:16.990 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-24T17:29:16.990 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a7981b84-025f-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-24T17:29:18.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:18 smithi032 bash[21457]: cluster 2024-04-24T17:29:17.035890+0000 mgr.y (mgr.14152) 54 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:18.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:18 smithi032 bash[21457]: cluster 2024-04-24T17:29:17.035890+0000 mgr.y (mgr.14152) 54 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:18.690 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/mon.b/config 2024-04-24T17:29:19.835 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:19.829+0000 7f17b8c57700 1 -- 172.21.15.82:0/3400021362 >> v1:172.21.15.32:6789/0 conn(0x7f17b4104060 legacy=0x7f17b4104440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:29:19.836 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:19.829+0000 7f17b8c57700 1 -- 172.21.15.82:0/3400021362 shutdown_connections 2024-04-24T17:29:19.836 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:19.829+0000 7f17b8c57700 1 -- 172.21.15.82:0/3400021362 >> 172.21.15.82:0/3400021362 conn(0x7f17b40fd8b0 msgr2=0x7f17b40ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:29:19.836 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:19.829+0000 7f17b8c57700 1 -- 172.21.15.82:0/3400021362 shutdown_connections 2024-04-24T17:29:19.836 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:19.829+0000 7f17b8c57700 1 -- 172.21.15.82:0/3400021362 wait complete. 2024-04-24T17:29:19.836 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:19.829+0000 7f17b8c57700 1 librados: starting msgr at 2024-04-24T17:29:19.836 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:19.829+0000 7f17b8c57700 1 librados: starting objecter 2024-04-24T17:29:19.836 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:19.829+0000 7f17b8c57700 1 Processor -- start 2024-04-24T17:29:19.836 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:19.833+0000 7f17b8c57700 1 -- start start 2024-04-24T17:29:19.836 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:19.833+0000 7f17b8c57700 1 librados: setting wanted keys 2024-04-24T17:29:19.836 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:19.833+0000 7f17b8c57700 1 librados: calling monclient init 2024-04-24T17:29:19.836 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:19.833+0000 7f17b8c57700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f17b4198930 con 0x7f17b4197e40 2024-04-24T17:29:19.838 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:19.833+0000 7f17b259c700 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f17b4197e40 0x7f17b4198220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.82:41380/0 (socket says 172.21.15.82:41380) 2024-04-24T17:29:19.838 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:19.833+0000 7f17b259c700 1 -- 172.21.15.82:0/3086341464 learned_addr learned my addr 172.21.15.82:0/3086341464 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-04-24T17:29:19.838 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:19.833+0000 7f17a37fe700 10 client.?.objecter ms_handle_connect 0x7f17b4197e40 2024-04-24T17:29:19.838 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:19.833+0000 7f17a37fe700 10 client.?.objecter resend_mon_ops 2024-04-24T17:29:19.838 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:19.833+0000 7f17a37fe700 1 -- 172.21.15.82:0/3086341464 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2402471204 0 0) 0x7f17b4198930 con 0x7f17b4197e40 2024-04-24T17:29:19.838 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:19.833+0000 7f17a37fe700 1 -- 172.21.15.82:0/3086341464 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f179c005730 con 0x7f17b4197e40 2024-04-24T17:29:19.838 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:19.833+0000 7f17a37fe700 1 -- 172.21.15.82:0/3086341464 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4225766652 0 0) 0x7f179c005730 con 0x7f17b4197e40 2024-04-24T17:29:19.838 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:19.833+0000 7f17a37fe700 1 -- 172.21.15.82:0/3086341464 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f17b4198930 con 0x7f17b4197e40 2024-04-24T17:29:19.838 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:19.833+0000 7f17a37fe700 1 -- 172.21.15.82:0/3086341464 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f17a40018e0 con 0x7f17b4197e40 2024-04-24T17:29:19.838 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:19.833+0000 7f17a37fe700 1 -- 172.21.15.82:0/3086341464 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2992532680 0 0) 0x7f17b4198930 con 0x7f17b4197e40 2024-04-24T17:29:19.839 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:19.833+0000 7f17a37fe700 1 -- 172.21.15.82:0/3086341464 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17b41a19b0 con 0x7f17b4197e40 2024-04-24T17:29:19.839 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:19.833+0000 7f17b8c57700 1 -- 172.21.15.82:0/3086341464 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f17b41a09a0 con 0x7f17b4197e40 2024-04-24T17:29:19.839 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:19.833+0000 7f17b8c57700 10 client.14190.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-24T17:29:19.839 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:19.833+0000 7f17b8c57700 1 -- 172.21.15.82:0/3086341464 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f17b41a0e60 con 0x7f17b4197e40 2024-04-24T17:29:19.839 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:19.833+0000 7f17b8c57700 1 librados: init done 2024-04-24T17:29:19.839 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:19.833+0000 7f17a37fe700 1 -- 172.21.15.82:0/3086341464 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(25 keys) ==== 1042+0+0 (unknown 2768897130 0 0) 0x7f17a4004700 con 0x7f17b4197e40 2024-04-24T17:29:19.839 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:19.833+0000 7f17a37fe700 1 -- 172.21.15.82:0/3086341464 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 2102506682 0 0) 0x7f17a4004b00 con 0x7f17b4197e40 2024-04-24T17:29:19.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:19.833+0000 7f17b8c57700 1 -- 172.21.15.82:0/3086341464 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1794005120 con 0x7f17b4197e40 2024-04-24T17:29:19.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:19.837+0000 7f17a37fe700 1 -- 172.21.15.82:0/3086341464 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 13) ==== 46579+0+0 (unknown 583022566 0 0) 0x7f17a4003890 con 0x7f17b4197e40 2024-04-24T17:29:19.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:19.837+0000 7f17a37fe700 1 -- 172.21.15.82:0/3086341464 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(4..4 src has 1..4) ==== 1069+0+0 (unknown 1280957937 0 0) 0x7f17a4047e40 con 0x7f17b4197e40 2024-04-24T17:29:19.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:19.837+0000 7f17a37fe700 10 client.14190.objecter ms_dispatch 0x7f17b4001550 osd_map(4..4 src has 1..4) 2024-04-24T17:29:19.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:19.837+0000 7f17a37fe700 3 client.14190.objecter handle_osd_map got epochs [4,4] > 0 2024-04-24T17:29:19.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:19.837+0000 7f17a37fe700 3 client.14190.objecter handle_osd_map decoding full epoch 4 2024-04-24T17:29:19.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:19.837+0000 7f17a37fe700 20 client.14190.objecter dump_active .. 0 homeless 2024-04-24T17:29:19.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:19.837+0000 7f17a37fe700 10 client.14190.objecter ms_handle_connect 0x7f179c03c2f0 2024-04-24T17:29:19.857 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:19.853+0000 7f17a37fe700 1 -- 172.21.15.82:0/3086341464 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188728 (unknown 1092875540 0 2656817166) 0x7f17a4016610 con 0x7f17b4197e40 2024-04-24T17:29:20.160 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:20.157+0000 7f17b8c57700 1 -- 172.21.15.82:0/3086341464 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f1794005410 con 0x7f17b4197e40 2024-04-24T17:29:20.164 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:20.161+0000 7f17a37fe700 1 -- 172.21.15.82:0/3086341464 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+694 (unknown 2237029548 0 3694938686) 0x7f17a4001bf0 con 0x7f17b4197e40 2024-04-24T17:29:20.164 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-24T17:29:20.164 INFO:teuthology.orchestra.run.smithi082.stdout:{"epoch":1,"fsid":"a7981b84-025f-11ef-bc93-c7b262605968","modified":"2024-04-24T17:26:59.605652Z","created":"2024-04-24T17:26:59.605652Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-24T17:29:20.166 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:20.161+0000 7f17b8c57700 10 librados: watch_flush enter 2024-04-24T17:29:20.166 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:20.161+0000 7f17b8c57700 10 librados: watch_flush exit 2024-04-24T17:29:20.166 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:20.161+0000 7f17b8c57700 20 client.14190.objecter shutdown clearing up homeless session... 2024-04-24T17:29:20.166 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:20.161+0000 7f17b8c57700 10 client.14190.objecter successfully canceled tick 2024-04-24T17:29:20.166 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:20.161+0000 7f17b8c57700 1 -- 172.21.15.82:0/3086341464 >> v1:172.21.15.32:6800/1847389373 conn(0x7f179c03c2f0 legacy=0x7f179c03e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:29:20.166 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:20.161+0000 7f17b8c57700 1 -- 172.21.15.82:0/3086341464 >> v1:172.21.15.32:6789/0 conn(0x7f17b4197e40 legacy=0x7f17b4198220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:29:20.166 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:20.161+0000 7f17b8c57700 1 -- 172.21.15.82:0/3086341464 shutdown_connections 2024-04-24T17:29:20.166 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:20.161+0000 7f17b8c57700 1 -- 172.21.15.82:0/3086341464 >> 172.21.15.82:0/3086341464 conn(0x7f17b40fd8b0 msgr2=0x7f17b4109920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:29:20.167 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:20.161+0000 7f17b8c57700 1 -- 172.21.15.82:0/3086341464 shutdown_connections 2024-04-24T17:29:20.167 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:20.161+0000 7f17b8c57700 1 -- 172.21.15.82:0/3086341464 wait complete. 2024-04-24T17:29:20.167 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:20.161+0000 7f17b8c57700 1 librados: shutdown 2024-04-24T17:29:20.167 INFO:teuthology.orchestra.run.smithi082.stderr:dumped monmap epoch 1 2024-04-24T17:29:20.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:20 smithi032 bash[21457]: cluster 2024-04-24T17:29:19.036804+0000 mgr.y (mgr.14152) 55 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:20.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:20 smithi032 bash[21457]: cluster 2024-04-24T17:29:19.036804+0000 mgr.y (mgr.14152) 55 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:20.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:20 smithi032 bash[21457]: audit 2024-04-24T17:29:19.735876+0000 mon.a (mon.0) 139 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:20.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:20 smithi032 bash[21457]: audit 2024-04-24T17:29:19.735876+0000 mon.a (mon.0) 139 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:20.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:20 smithi032 bash[21457]: audit 2024-04-24T17:29:19.781566+0000 mon.a (mon.0) 140 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:20.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:20 smithi032 bash[21457]: audit 2024-04-24T17:29:19.781566+0000 mon.a (mon.0) 140 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:20.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:20 smithi032 bash[21457]: audit 2024-04-24T17:29:19.783029+0000 mon.a (mon.0) 141 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T17:29:20.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:20 smithi032 bash[21457]: audit 2024-04-24T17:29:19.783029+0000 mon.a (mon.0) 141 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T17:29:20.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:20 smithi032 bash[21457]: audit 2024-04-24T17:29:19.784673+0000 mon.a (mon.0) 142 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:20.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:20 smithi032 bash[21457]: audit 2024-04-24T17:29:19.784673+0000 mon.a (mon.0) 142 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:21.173 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:21 smithi082 bash[21793]: cluster 2024-04-24T17:29:15.033644+0000 mgr.y (mgr.14152) 51 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:21.173 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:21 smithi082 bash[21793]: cluster 2024-04-24T17:29:15.034964+0000 mgr.y (mgr.14152) 52 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:21.173 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:21 smithi082 bash[21793]: cluster 2024-04-24T17:29:15.034964+0000 mgr.y (mgr.14152) 52 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:21.173 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:21 smithi082 bash[21793]: cephadm 2024-04-24T17:29:15.048701+0000 mgr.y (mgr.14152) 53 : cephadm [INF] Deploying daemon mon.b on smithi082 2024-04-24T17:29:21.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:21 smithi082 bash[21793]: cephadm 2024-04-24T17:29:15.048701+0000 mgr.y (mgr.14152) 53 : cephadm [INF] Deploying daemon mon.b on smithi082 2024-04-24T17:29:21.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:21 smithi082 bash[21793]: audit 2024-04-24T17:29:15.155067+0000 mon.a (mon.0) 138 : audit [DBG] from='client.? v1:172.21.15.82:0/2880720608' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T17:29:21.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:21 smithi082 bash[21793]: audit 2024-04-24T17:29:15.155067+0000 mon.a (mon.0) 138 : audit [DBG] from='client.? v1:172.21.15.82:0/2880720608' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T17:29:21.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:21 smithi082 bash[21793]: cluster 2024-04-24T17:29:17.035890+0000 mgr.y (mgr.14152) 54 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:21.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:21 smithi082 bash[21793]: cluster 2024-04-24T17:29:17.035890+0000 mgr.y (mgr.14152) 54 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:21.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:21 smithi082 bash[21793]: cluster 2024-04-24T17:29:19.036804+0000 mgr.y (mgr.14152) 55 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:21.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:21 smithi082 bash[21793]: cluster 2024-04-24T17:29:19.036804+0000 mgr.y (mgr.14152) 55 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:21.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:21 smithi082 bash[21793]: audit 2024-04-24T17:29:19.735876+0000 mon.a (mon.0) 139 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:21.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:21 smithi082 bash[21793]: audit 2024-04-24T17:29:19.735876+0000 mon.a (mon.0) 139 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:21.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:21 smithi082 bash[21793]: audit 2024-04-24T17:29:19.781566+0000 mon.a (mon.0) 140 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:21.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:21 smithi082 bash[21793]: audit 2024-04-24T17:29:19.781566+0000 mon.a (mon.0) 140 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:21.381 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:21 smithi082 bash[21793]: audit 2024-04-24T17:29:19.783029+0000 mon.a (mon.0) 141 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T17:29:21.381 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:21 smithi082 bash[21793]: audit 2024-04-24T17:29:19.783029+0000 mon.a (mon.0) 141 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T17:29:21.381 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:21 smithi082 bash[21793]: audit 2024-04-24T17:29:19.784673+0000 mon.a (mon.0) 142 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:21.381 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:21 smithi082 bash[21793]: audit 2024-04-24T17:29:19.784673+0000 mon.a (mon.0) 142 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:21.381 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:21 smithi082 bash[21793]: debug 2024-04-24T17:29:21.161+0000 7f3118140700 1 mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-24T17:29:21.437 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:21 smithi032 bash[21457]: cephadm 2024-04-24T17:29:19.786666+0000 mgr.y (mgr.14152) 56 : cephadm [INF] Deploying daemon mon.c on smithi032 2024-04-24T17:29:21.437 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:21 smithi032 bash[21457]: cephadm 2024-04-24T17:29:19.786666+0000 mgr.y (mgr.14152) 56 : cephadm [INF] Deploying daemon mon.c on smithi032 2024-04-24T17:29:21.437 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:21 smithi032 bash[21457]: audit 2024-04-24T17:29:20.164933+0000 mon.a (mon.0) 143 : audit [DBG] from='client.? v1:172.21.15.82:0/3086341464' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T17:29:21.438 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:21 smithi032 bash[21457]: audit 2024-04-24T17:29:20.164933+0000 mon.a (mon.0) 143 : audit [DBG] from='client.? v1:172.21.15.82:0/3086341464' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T17:29:22.013 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-24T17:29:22.013 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a7981b84-025f-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-24T17:29:23.691 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:29:23 smithi032 systemd[1]: /etc/systemd/system/ceph-a7981b84-025f-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-24T17:29:23.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:23 smithi032 systemd[1]: /etc/systemd/system/ceph-a7981b84-025f-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-24T17:29:23.966 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:23 smithi032 systemd[1]: /etc/systemd/system/ceph-a7981b84-025f-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-24T17:29:23.967 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:29:23 smithi032 systemd[1]: /etc/systemd/system/ceph-a7981b84-025f-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-24T17:29:23.967 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:23 smithi032 systemd[1]: Started Ceph mon.c for a7981b84-025f-11ef-bc93-c7b262605968. 2024-04-24T17:29:25.253 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.950+0000 7f1685391c80 0 set uid:gid to 167:167 (ceph:ceph) 2024-04-24T17:29:25.253 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.950+0000 7f1685391c80 0 ceph version 19.0.0-3285-g5904d294 (5904d29475f5be602879d9fb26280e89b808d5cc) squid (dev), process ceph-mon, pid 7 2024-04-24T17:29:25.253 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.950+0000 7f1685391c80 0 pidfile_write: ignore empty --pid-file 2024-04-24T17:29:25.253 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 0 load: jerasure load: lrc 2024-04-24T17:29:25.253 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: RocksDB version: 7.9.2 2024-04-24T17:29:25.253 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Git sha 0 2024-04-24T17:29:25.254 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Compile date 2024-04-24 13:56:12 2024-04-24T17:29:25.254 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: DB SUMMARY 2024-04-24T17:29:25.254 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: DB Session ID: ALRHCRHOH7RCAYFGITWW 2024-04-24T17:29:25.254 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: CURRENT file: CURRENT 2024-04-24T17:29:25.254 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: IDENTITY file: IDENTITY 2024-04-24T17:29:25.254 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-24T17:29:25.254 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-04-24T17:29:25.254 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 476 ; 2024-04-24T17:29:25.254 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.error_if_exists: 0 2024-04-24T17:29:25.254 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.create_if_missing: 0 2024-04-24T17:29:25.254 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.paranoid_checks: 1 2024-04-24T17:29:25.254 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.flush_verify_memtable_count: 1 2024-04-24T17:29:25.255 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-24T17:29:25.255 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-24T17:29:25.255 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.env: 0x55fec4d38cc0 2024-04-24T17:29:25.255 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.fs: PosixFileSystem 2024-04-24T17:29:25.255 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.info_log: 0x55fec5621e20 2024-04-24T17:29:25.255 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.max_file_opening_threads: 16 2024-04-24T17:29:25.255 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.statistics: (nil) 2024-04-24T17:29:25.255 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.use_fsync: 0 2024-04-24T17:29:25.255 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.max_log_file_size: 0 2024-04-24T17:29:25.256 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-24T17:29:25.256 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.log_file_time_to_roll: 0 2024-04-24T17:29:25.256 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.keep_log_file_num: 1000 2024-04-24T17:29:25.256 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.recycle_log_file_num: 0 2024-04-24T17:29:25.256 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.allow_fallocate: 1 2024-04-24T17:29:25.256 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.allow_mmap_reads: 0 2024-04-24T17:29:25.256 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.allow_mmap_writes: 0 2024-04-24T17:29:25.256 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.use_direct_reads: 0 2024-04-24T17:29:25.256 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-24T17:29:25.256 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.create_missing_column_families: 0 2024-04-24T17:29:25.256 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.db_log_dir: 2024-04-24T17:29:25.256 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.wal_dir: 2024-04-24T17:29:25.257 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.table_cache_numshardbits: 6 2024-04-24T17:29:25.257 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-04-24T17:29:25.257 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-04-24T17:29:25.257 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-24T17:29:25.257 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-24T17:29:25.257 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-04-24T17:29:25.257 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.advise_random_on_open: 1 2024-04-24T17:29:25.257 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.db_write_buffer_size: 0 2024-04-24T17:29:25.257 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.write_buffer_manager: 0x55fec55f30e0 2024-04-24T17:29:25.257 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-24T17:29:25.257 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-24T17:29:25.257 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.use_adaptive_mutex: 0 2024-04-24T17:29:25.257 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.rate_limiter: (nil) 2024-04-24T17:29:25.257 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-24T17:29:25.258 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.wal_recovery_mode: 2 2024-04-24T17:29:25.258 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.enable_thread_tracking: 0 2024-04-24T17:29:25.258 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.enable_pipelined_write: 0 2024-04-24T17:29:25.258 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.unordered_write: 0 2024-04-24T17:29:25.258 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-24T17:29:25.258 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-24T17:29:25.258 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-24T17:29:25.258 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-24T17:29:25.258 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.row_cache: None 2024-04-24T17:29:25.258 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.wal_filter: None 2024-04-24T17:29:25.258 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-24T17:29:25.258 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.allow_ingest_behind: 0 2024-04-24T17:29:25.258 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.two_write_queues: 0 2024-04-24T17:29:25.259 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.manual_wal_flush: 0 2024-04-24T17:29:25.259 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.wal_compression: 0 2024-04-24T17:29:25.259 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.atomic_flush: 0 2024-04-24T17:29:25.259 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-24T17:29:25.259 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.persist_stats_to_disk: 0 2024-04-24T17:29:25.259 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-04-24T17:29:25.259 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.log_readahead_size: 0 2024-04-24T17:29:25.259 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-24T17:29:25.259 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.best_efforts_recovery: 0 2024-04-24T17:29:25.259 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-24T17:29:25.259 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-24T17:29:25.259 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.allow_data_in_errors: 0 2024-04-24T17:29:25.259 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.db_host_id: __hostname__ 2024-04-24T17:29:25.259 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.enforce_single_del_contracts: true 2024-04-24T17:29:25.260 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.max_background_jobs: 2 2024-04-24T17:29:25.260 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.max_background_compactions: -1 2024-04-24T17:29:25.260 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.max_subcompactions: 1 2024-04-24T17:29:25.260 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-24T17:29:25.260 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-24T17:29:25.260 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.delayed_write_rate : 16777216 2024-04-24T17:29:25.260 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.max_total_wal_size: 0 2024-04-24T17:29:25.260 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-24T17:29:25.260 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.stats_dump_period_sec: 600 2024-04-24T17:29:25.260 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.stats_persist_period_sec: 600 2024-04-24T17:29:25.260 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-24T17:29:25.260 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.max_open_files: -1 2024-04-24T17:29:25.260 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.bytes_per_sync: 0 2024-04-24T17:29:25.260 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-04-24T17:29:25.261 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-04-24T17:29:25.261 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.compaction_readahead_size: 0 2024-04-24T17:29:25.261 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.max_background_flushes: -1 2024-04-24T17:29:25.261 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Compression algorithms supported: 2024-04-24T17:29:25.261 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-24T17:29:25.261 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: kZSTD supported: 0 2024-04-24T17:29:25.261 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: kXpressCompression supported: 0 2024-04-24T17:29:25.261 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: kLZ4HCCompression supported: 1 2024-04-24T17:29:25.261 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: kZlibCompression supported: 1 2024-04-24T17:29:25.261 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: kSnappyCompression supported: 1 2024-04-24T17:29:25.261 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: kLZ4Compression supported: 1 2024-04-24T17:29:25.261 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: kBZip2Compression supported: 0 2024-04-24T17:29:25.261 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-04-24T17:29:25.261 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: DMutex implementation: pthread_mutex_t 2024-04-24T17:29:25.261 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-04-24T17:29:25.262 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-24T17:29:25.263 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-24T17:29:25.263 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.merge_operator: 2024-04-24T17:29:25.263 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.compaction_filter: None 2024-04-24T17:29:25.263 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.compaction_filter_factory: None 2024-04-24T17:29:25.264 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.sst_partitioner_factory: None 2024-04-24T17:29:25.264 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-04-24T17:29:25.264 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.table_factory: BlockBasedTable 2024-04-24T17:29:25.264 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55fec5621fe0) 2024-04-24T17:29:25.264 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cache_index_and_filter_blocks: 1 2024-04-24T17:29:25.264 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-24T17:29:25.264 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-24T17:29:25.264 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: pin_top_level_index_and_filter: 1 2024-04-24T17:29:25.264 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: index_type: 0 2024-04-24T17:29:25.264 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: data_block_index_type: 0 2024-04-24T17:29:25.264 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: index_shortening: 1 2024-04-24T17:29:25.264 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: data_block_hash_table_util_ratio: 0.750000 2024-04-24T17:29:25.264 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: checksum: 4 2024-04-24T17:29:25.264 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: no_block_cache: 0 2024-04-24T17:29:25.265 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: block_cache: 0x55fec56231f0 2024-04-24T17:29:25.265 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: block_cache_name: BinnedLRUCache 2024-04-24T17:29:25.265 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: block_cache_options: 2024-04-24T17:29:25.265 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: capacity : 536870912 2024-04-24T17:29:25.265 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: num_shard_bits : 4 2024-04-24T17:29:25.265 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: strict_capacity_limit : 0 2024-04-24T17:29:25.265 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: high_pri_pool_ratio: 0.000 2024-04-24T17:29:25.265 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: block_cache_compressed: (nil) 2024-04-24T17:29:25.265 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: persistent_cache: (nil) 2024-04-24T17:29:25.265 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: block_size: 4096 2024-04-24T17:29:25.265 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: block_size_deviation: 10 2024-04-24T17:29:25.265 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: block_restart_interval: 16 2024-04-24T17:29:25.265 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: index_block_restart_interval: 1 2024-04-24T17:29:25.265 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: metadata_block_size: 4096 2024-04-24T17:29:25.265 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: partition_filters: 0 2024-04-24T17:29:25.266 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: use_delta_encoding: 1 2024-04-24T17:29:25.266 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: filter_policy: bloomfilter 2024-04-24T17:29:25.266 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: whole_key_filtering: 1 2024-04-24T17:29:25.266 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: verify_compression: 0 2024-04-24T17:29:25.266 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: read_amp_bytes_per_bit: 0 2024-04-24T17:29:25.266 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: format_version: 5 2024-04-24T17:29:25.266 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: enable_index_compression: 1 2024-04-24T17:29:25.266 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: block_align: 0 2024-04-24T17:29:25.266 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: max_auto_readahead_size: 262144 2024-04-24T17:29:25.266 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: prepopulate_block_cache: 0 2024-04-24T17:29:25.266 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: initial_auto_readahead_size: 8192 2024-04-24T17:29:25.266 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: num_file_reads_for_auto_readahead: 2 2024-04-24T17:29:25.266 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.write_buffer_size: 33554432 2024-04-24T17:29:25.267 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.max_write_buffer_number: 2 2024-04-24T17:29:25.267 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.compression: NoCompression 2024-04-24T17:29:25.267 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.bottommost_compression: Disabled 2024-04-24T17:29:25.267 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.prefix_extractor: nullptr 2024-04-24T17:29:25.267 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-24T17:29:25.267 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.num_levels: 7 2024-04-24T17:29:25.267 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-24T17:29:25.267 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-24T17:29:25.267 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-24T17:29:25.267 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-24T17:29:25.267 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-24T17:29:25.267 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-24T17:29:25.267 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-24T17:29:25.267 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-24T17:29:25.267 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-24T17:29:25.268 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-24T17:29:25.268 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-24T17:29:25.268 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-24T17:29:25.268 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.compression_opts.window_bits: -14 2024-04-24T17:29:25.268 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.compression_opts.level: 32767 2024-04-24T17:29:25.268 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.compression_opts.strategy: 0 2024-04-24T17:29:25.268 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-24T17:29:25.268 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-24T17:29:25.268 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-24T17:29:25.268 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-24T17:29:25.268 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.compression_opts.enabled: false 2024-04-24T17:29:25.268 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-24T17:29:25.268 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-24T17:29:25.268 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-24T17:29:25.268 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-24T17:29:25.269 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.target_file_size_base: 67108864 2024-04-24T17:29:25.269 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.target_file_size_multiplier: 1 2024-04-24T17:29:25.269 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-24T17:29:25.269 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-24T17:29:25.269 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-24T17:29:25.269 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-24T17:29:25.269 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-24T17:29:25.269 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-24T17:29:25.269 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-24T17:29:25.269 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-24T17:29:25.269 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-24T17:29:25.269 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-24T17:29:25.269 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-24T17:29:25.269 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-24T17:29:25.269 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-24T17:29:25.270 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.arena_block_size: 1048576 2024-04-24T17:29:25.270 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-24T17:29:25.270 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-24T17:29:25.270 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.disable_auto_compactions: 0 2024-04-24T17:29:25.270 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-24T17:29:25.270 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-24T17:29:25.270 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-24T17:29:25.270 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-24T17:29:25.270 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-24T17:29:25.270 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-24T17:29:25.270 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-24T17:29:25.270 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-24T17:29:25.270 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-24T17:29:25.270 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-24T17:29:25.271 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-24T17:29:25.271 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.inplace_update_support: 0 2024-04-24T17:29:25.271 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-04-24T17:29:25.271 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-24T17:29:25.271 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-24T17:29:25.271 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.memtable_huge_page_size: 0 2024-04-24T17:29:25.271 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.bloom_locality: 0 2024-04-24T17:29:25.271 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.max_successive_merges: 0 2024-04-24T17:29:25.271 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-04-24T17:29:25.271 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.paranoid_file_checks: 0 2024-04-24T17:29:25.271 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.force_consistency_checks: 1 2024-04-24T17:29:25.271 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.report_bg_io_stats: 0 2024-04-24T17:29:25.271 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.ttl: 2592000 2024-04-24T17:29:25.271 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-04-24T17:29:25.272 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-24T17:29:25.272 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-24T17:29:25.272 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.enable_blob_files: false 2024-04-24T17:29:25.272 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.min_blob_size: 0 2024-04-24T17:29:25.272 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.blob_file_size: 268435456 2024-04-24T17:29:25.272 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.blob_compression_type: NoCompression 2024-04-24T17:29:25.272 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.enable_blob_garbage_collection: false 2024-04-24T17:29:25.272 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-24T17:29:25.272 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-24T17:29:25.272 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-24T17:29:25.272 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.blob_file_starting_level: 0 2024-04-24T17:29:25.272 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.954+0000 7f1685391c80 4 rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-24T17:29:25.272 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.958+0000 7f1685391c80 4 rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-24T17:29:25.272 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.958+0000 7f1685391c80 4 rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-24T17:29:25.272 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.958+0000 7f1685391c80 4 rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 8bea2976-0059-4b87-a837-82741a50e643 2024-04-24T17:29:25.273 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.958+0000 7f1685391c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1713979764959454, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-24T17:29:25.273 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.958+0000 7f1685391c80 4 rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-24T17:29:25.273 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.958+0000 7f1685391c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1713979764960004, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1613, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 488, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 366, "raw_average_value_size": 73, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713979764, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "8bea2976-0059-4b87-a837-82741a50e643", "db_session_id": "ALRHCRHOH7RCAYFGITWW", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-24T17:29:25.273 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.958+0000 7f1685391c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1713979764960131, "job": 1, "event": "recovery_finished"} 2024-04-24T17:29:25.273 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.958+0000 7f1685391c80 4 rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-24T17:29:25.273 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.958+0000 7f1685391c80 4 rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-24T17:29:25.273 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.958+0000 7f1685391c80 4 rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55fec5744000 2024-04-24T17:29:25.273 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.958+0000 7f1685391c80 4 rocksdb: DB pointer 0x55fec5730000 2024-04-24T17:29:25.273 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.958+0000 7f1673ae9700 4 rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-24T17:29:25.273 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.958+0000 7f1673ae9700 4 rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-24T17:29:25.273 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: ** DB Stats ** 2024-04-24T17:29:25.273 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: Uptime(secs): 0.0 total, 0.0 interval 2024-04-24T17:29:25.273 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-24T17:29:25.273 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-24T17:29:25.274 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-24T17:29:25.274 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-24T17:29:25.274 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-24T17:29:25.274 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-24T17:29:25.274 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: ** Compaction Stats [default] ** 2024-04-24T17:29:25.274 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-24T17:29:25.274 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-24T17:29:25.274 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-24T17:29:25.274 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-24T17:29:25.274 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-24T17:29:25.274 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: ** Compaction Stats [default] ** 2024-04-24T17:29:25.274 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-24T17:29:25.274 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-24T17:29:25.274 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-24T17:29:25.275 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-24T17:29:25.275 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: Uptime(secs): 0.0 total, 0.0 interval 2024-04-24T17:29:25.275 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: Flush(GB): cumulative 0.000, interval 0.000 2024-04-24T17:29:25.275 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-24T17:29:25.275 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: AddFile(Total Files): cumulative 0, interval 0 2024-04-24T17:29:25.275 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: AddFile(L0 Files): cumulative 0, interval 0 2024-04-24T17:29:25.275 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: AddFile(Keys): cumulative 0, interval 0 2024-04-24T17:29:25.275 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: Cumulative compaction: 0.00 GB write, 0.44 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-24T17:29:25.275 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: Interval compaction: 0.00 GB write, 0.44 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-24T17:29:25.275 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-24T17:29:25.275 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: Block cache BinnedLRUCache@0x55fec56231f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-04-24T17:29:25.275 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-24T17:29:25.275 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: ** File Read Latency Histogram By Level [default] ** 2024-04-24T17:29:25.275 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.958+0000 7f1685391c80 0 mon.c does not exist in monmap, will attempt to join an existing cluster 2024-04-24T17:29:25.275 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.958+0000 7f1685391c80 0 using public_addrv v1:172.21.15.32:6790/0 2024-04-24T17:29:25.276 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.958+0000 7f1685391c80 0 starting mon.c rank -1 at public addrs v1:172.21.15.32:6790/0 at bind addrs v1:172.21.15.32:6790/0 mon_data /var/lib/ceph/mon/ceph-c fsid a7981b84-025f-11ef-bc93-c7b262605968 2024-04-24T17:29:25.276 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.962+0000 7f1685391c80 1 mon.c@-1(???) e0 preinit fsid a7981b84-025f-11ef-bc93-c7b262605968 2024-04-24T17:29:25.276 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.978+0000 7f16722e6700 4 rocksdb: [db/db_impl/db_impl_write.cc:2098] [default] New memtable created with log file: #13. Immutable memtables: 0. 2024-04-24T17:29:25.276 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.978+0000 7f167b2f8700 4 rocksdb: (Original Log Time 2024/04/24-17:29:24.979334) [db/db_impl/db_impl_compaction_flush.cc:2832] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 2024-04-24T17:29:25.276 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.978+0000 7f167b2f8700 4 rocksdb: [db/flush_job.cc:856] [default] [JOB 3] Flushing memtable with next log file: 13 2024-04-24T17:29:25.276 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.978+0000 7f167b2f8700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1713979764979395, "job": 3, "event": "flush_started", "num_memtables": 1, "num_entries": 1139, "num_deletes": 20, "total_data_size": 2560052, "memory_usage": 2583008, "flush_reason": "Manual Compaction"} 2024-04-24T17:29:25.276 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.978+0000 7f167b2f8700 4 rocksdb: [db/flush_job.cc:885] [default] [JOB 3] Level-0 flush table #14: started 2024-04-24T17:29:25.276 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.978+0000 7f1676aef700 0 mon.c@-1(synchronizing).mds e1 new map 2024-04-24T17:29:25.276 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.978+0000 7f1676aef700 0 mon.c@-1(synchronizing).mds e1 print_map 2024-04-24T17:29:25.276 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: e1 2024-04-24T17:29:25.276 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: enable_multiple, ever_enabled_multiple: 1,1 2024-04-24T17:29:25.276 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-24T17:29:25.276 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: legacy client fscid: -1 2024-04-24T17:29:25.276 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: 2024-04-24T17:29:25.276 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: No filesystems configured 2024-04-24T17:29:25.277 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.978+0000 7f1676aef700 1 mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-24T17:29:25.277 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.978+0000 7f1676aef700 1 mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-24T17:29:25.277 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.978+0000 7f1676aef700 1 mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-24T17:29:25.277 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.978+0000 7f1676aef700 1 mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-24T17:29:25.277 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.978+0000 7f1676aef700 1 mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-24T17:29:25.277 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.978+0000 7f1676aef700 1 mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-24T17:29:25.277 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.978+0000 7f1676aef700 0 mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-24T17:29:25.277 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.978+0000 7f1676aef700 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-24T17:29:25.277 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.978+0000 7f1676aef700 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-24T17:29:25.277 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.978+0000 7f1676aef700 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-24T17:29:25.277 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:28:38.753391+0000 mgr.y (mgr.14152) 14 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.277 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:28:38.753391+0000 mgr.y (mgr.14152) 14 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.277 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:28:40.977324+0000 mon.a (mon.0) 108 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:25.277 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:28:40.977324+0000 mon.a (mon.0) 108 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:25.277 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:28:40.979224+0000 mon.a (mon.0) 109 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T17:29:25.278 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:28:40.979224+0000 mon.a (mon.0) 109 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T17:29:25.278 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:28:40.754045+0000 mgr.y (mgr.14152) 15 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.278 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:28:40.754045+0000 mgr.y (mgr.14152) 15 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.278 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cephadm 2024-04-24T17:28:40.981988+0000 mgr.y (mgr.14152) 16 : cephadm [INF] Updating smithi032:/etc/ceph/ceph.conf 2024-04-24T17:29:25.278 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cephadm 2024-04-24T17:28:40.981988+0000 mgr.y (mgr.14152) 16 : cephadm [INF] Updating smithi032:/etc/ceph/ceph.conf 2024-04-24T17:29:25.278 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cephadm 2024-04-24T17:28:41.198285+0000 mgr.y (mgr.14152) 17 : cephadm [INF] Updating smithi032:/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T17:29:25.278 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cephadm 2024-04-24T17:28:41.198285+0000 mgr.y (mgr.14152) 17 : cephadm [INF] Updating smithi032:/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T17:29:25.278 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cephadm 2024-04-24T17:28:41.312660+0000 mgr.y (mgr.14152) 18 : cephadm [INF] Updating smithi032:/etc/ceph/ceph.client.admin.keyring 2024-04-24T17:29:25.278 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cephadm 2024-04-24T17:28:41.312660+0000 mgr.y (mgr.14152) 18 : cephadm [INF] Updating smithi032:/etc/ceph/ceph.client.admin.keyring 2024-04-24T17:29:25.278 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:28:41.545473+0000 mon.a (mon.0) 110 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:25.278 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:28:41.545473+0000 mon.a (mon.0) 110 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:25.278 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:28:41.549104+0000 mon.a (mon.0) 111 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:25.278 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:28:41.549104+0000 mon.a (mon.0) 111 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:25.278 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:28:41.582459+0000 mon.a (mon.0) 112 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T17:29:25.279 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:28:41.582459+0000 mon.a (mon.0) 112 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T17:29:25.279 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cephadm 2024-04-24T17:28:41.426976+0000 mgr.y (mgr.14152) 19 : cephadm [INF] Updating smithi032:/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T17:29:25.279 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cephadm 2024-04-24T17:28:41.426976+0000 mgr.y (mgr.14152) 19 : cephadm [INF] Updating smithi032:/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T17:29:25.279 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:28:41.546599+0000 mgr.y (mgr.14152) 20 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.279 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:28:41.546599+0000 mgr.y (mgr.14152) 20 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.279 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:28:41.547200+0000 mgr.y (mgr.14152) 21 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.279 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:28:41.547200+0000 mgr.y (mgr.14152) 21 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.279 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:28:43.547960+0000 mgr.y (mgr.14152) 22 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.279 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:28:43.547960+0000 mgr.y (mgr.14152) 22 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.279 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:28:44.280687+0000 mgr.y (mgr.14152) 23 : audit [DBG] from='client.14178 v1:172.21.15.32:0/3686357677' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi082", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T17:29:25.279 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:28:44.280687+0000 mgr.y (mgr.14152) 23 : audit [DBG] from='client.14178 v1:172.21.15.32:0/3686357677' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi082", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T17:29:25.279 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cephadm 2024-04-24T17:28:44.996769+0000 mgr.y (mgr.14152) 24 : cephadm [INF] Deploying cephadm binary to smithi082 2024-04-24T17:29:25.279 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cephadm 2024-04-24T17:28:44.996769+0000 mgr.y (mgr.14152) 24 : cephadm [INF] Deploying cephadm binary to smithi082 2024-04-24T17:29:25.279 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:28:45.548691+0000 mgr.y (mgr.14152) 25 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.280 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:28:45.548691+0000 mgr.y (mgr.14152) 25 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.280 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:28:47.549433+0000 mgr.y (mgr.14152) 26 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.280 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:28:47.549433+0000 mgr.y (mgr.14152) 26 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.280 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:28:49.360105+0000 mon.a (mon.0) 113 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:25.280 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:28:49.360105+0000 mon.a (mon.0) 113 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:25.280 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:28:49.383711+0000 mon.a (mon.0) 114 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T17:29:25.280 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:28:49.383711+0000 mon.a (mon.0) 114 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T17:29:25.280 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:28:49.394534+0000 mon.a (mon.0) 115 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:25.280 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:28:49.394534+0000 mon.a (mon.0) 115 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:25.280 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:28:49.389199+0000 mgr.y (mgr.14152) 27 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.280 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:28:49.389199+0000 mgr.y (mgr.14152) 27 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.280 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:28:49.389894+0000 mgr.y (mgr.14152) 28 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.280 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:28:49.389894+0000 mgr.y (mgr.14152) 28 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.280 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:28:50.480686+0000 mon.a (mon.0) 116 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:25.280 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:28:50.480686+0000 mon.a (mon.0) 116 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:25.281 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:28:50.516262+0000 mon.a (mon.0) 117 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T17:29:25.281 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:28:50.516262+0000 mon.a (mon.0) 117 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T17:29:25.281 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:28:50.987627+0000 mon.a (mon.0) 118 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:25.281 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:28:50.987627+0000 mon.a (mon.0) 118 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:25.281 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cephadm 2024-04-24T17:28:50.481597+0000 mgr.y (mgr.14152) 29 : cephadm [INF] Added host smithi082 2024-04-24T17:29:25.281 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cephadm 2024-04-24T17:28:50.481597+0000 mgr.y (mgr.14152) 29 : cephadm [INF] Added host smithi082 2024-04-24T17:29:25.281 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:28:51.390558+0000 mgr.y (mgr.14152) 30 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.281 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:28:51.390558+0000 mgr.y (mgr.14152) 30 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.281 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:28:52.461457+0000 mon.a (mon.0) 119 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:25.281 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:28:52.461457+0000 mon.a (mon.0) 119 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:25.282 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:28:53.391312+0000 mgr.y (mgr.14152) 31 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.282 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:28:53.391312+0000 mgr.y (mgr.14152) 31 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.282 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:28:53.603070+0000 mon.a (mon.0) 120 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:25.282 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:28:53.603070+0000 mon.a (mon.0) 120 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:25.282 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:28:55.392000+0000 mgr.y (mgr.14152) 32 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.283 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:28:55.392000+0000 mgr.y (mgr.14152) 32 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.283 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:28:56.830405+0000 mon.a (mon.0) 121 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:25.283 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:28:56.830405+0000 mon.a (mon.0) 121 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:25.283 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:28:56.831624+0000 mon.a (mon.0) 122 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi082", "name": "osd_memory_target"}]: dispatch 2024-04-24T17:29:25.283 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:28:56.831624+0000 mon.a (mon.0) 122 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi082", "name": "osd_memory_target"}]: dispatch 2024-04-24T17:29:25.283 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:28:57.180678+0000 mgr.y (mgr.14152) 33 : audit [DBG] from='client.14180 v1:172.21.15.32:0/3175493670' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T17:29:25.283 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:28:57.180678+0000 mgr.y (mgr.14152) 33 : audit [DBG] from='client.14180 v1:172.21.15.32:0/3175493670' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T17:29:25.283 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:28:57.392705+0000 mgr.y (mgr.14152) 34 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.283 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:28:57.392705+0000 mgr.y (mgr.14152) 34 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.283 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:28:58.109403+0000 mon.a (mon.0) 123 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:25.283 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:28:58.109403+0000 mon.a (mon.0) 123 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:25.283 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:28:58.111049+0000 mon.a (mon.0) 124 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T17:29:25.283 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:28:58.111049+0000 mon.a (mon.0) 124 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T17:29:25.283 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cephadm 2024-04-24T17:28:58.113585+0000 mgr.y (mgr.14152) 35 : cephadm [INF] Updating smithi082:/etc/ceph/ceph.conf 2024-04-24T17:29:25.284 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cephadm 2024-04-24T17:28:58.113585+0000 mgr.y (mgr.14152) 35 : cephadm [INF] Updating smithi082:/etc/ceph/ceph.conf 2024-04-24T17:29:25.284 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cephadm 2024-04-24T17:28:58.328659+0000 mgr.y (mgr.14152) 36 : cephadm [INF] Updating smithi082:/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T17:29:25.284 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cephadm 2024-04-24T17:28:58.328659+0000 mgr.y (mgr.14152) 36 : cephadm [INF] Updating smithi082:/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T17:29:25.284 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:28:58.829024+0000 mon.a (mon.0) 125 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:25.284 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:28:58.829024+0000 mon.a (mon.0) 125 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:25.284 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cephadm 2024-04-24T17:28:58.493807+0000 mgr.y (mgr.14152) 37 : cephadm [INF] Updating smithi082:/etc/ceph/ceph.client.admin.keyring 2024-04-24T17:29:25.284 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cephadm 2024-04-24T17:28:58.493807+0000 mgr.y (mgr.14152) 37 : cephadm [INF] Updating smithi082:/etc/ceph/ceph.client.admin.keyring 2024-04-24T17:29:25.284 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cephadm 2024-04-24T17:28:58.658745+0000 mgr.y (mgr.14152) 38 : cephadm [INF] Updating smithi082:/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T17:29:25.284 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cephadm 2024-04-24T17:28:58.658745+0000 mgr.y (mgr.14152) 38 : cephadm [INF] Updating smithi082:/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T17:29:25.284 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:28:58.830203+0000 mgr.y (mgr.14152) 39 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.284 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:28:58.830203+0000 mgr.y (mgr.14152) 39 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.284 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:28:58.831393+0000 mgr.y (mgr.14152) 40 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.284 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:28:58.831393+0000 mgr.y (mgr.14152) 40 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.284 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:28:58.836349+0000 mon.a (mon.0) 126 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:25.284 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:28:58.836349+0000 mon.a (mon.0) 126 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:25.284 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:29:00.831955+0000 mgr.y (mgr.14152) 41 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.285 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:29:00.831955+0000 mgr.y (mgr.14152) 41 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.285 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:29:02.832768+0000 mgr.y (mgr.14152) 42 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.285 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:29:02.832768+0000 mgr.y (mgr.14152) 42 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.285 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:29:04.107358+0000 mon.a (mon.0) 127 : audit [INF] from='client.? v1:172.21.15.32:0/3025043927' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "jewel"}]: dispatch 2024-04-24T17:29:25.285 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:29:04.107358+0000 mon.a (mon.0) 127 : audit [INF] from='client.? v1:172.21.15.32:0/3025043927' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "jewel"}]: dispatch 2024-04-24T17:29:25.285 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:29:04.833787+0000 mgr.y (mgr.14152) 43 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.285 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:29:04.833787+0000 mgr.y (mgr.14152) 43 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.285 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:29:04.848534+0000 mon.a (mon.0) 128 : audit [INF] from='client.? v1:172.21.15.32:0/3025043927' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "jewel"}]': finished 2024-04-24T17:29:25.285 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:29:04.848534+0000 mon.a (mon.0) 128 : audit [INF] from='client.? v1:172.21.15.32:0/3025043927' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "jewel"}]': finished 2024-04-24T17:29:25.285 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:29:04.853268+0000 mon.a (mon.0) 129 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-04-24T17:29:25.285 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:29:04.853268+0000 mon.a (mon.0) 129 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-04-24T17:29:25.285 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:29:06.834835+0000 mgr.y (mgr.14152) 44 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.285 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:29:06.834835+0000 mgr.y (mgr.14152) 44 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.285 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:29:07.966095+0000 mgr.y (mgr.14152) 45 : audit [DBG] from='client.14184 v1:172.21.15.82:0/3357327228' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi032:[v1:172.21.15.32:6789]=a;smithi032:[v1:172.21.15.32:6790]=c;smithi082:[v1:172.21.15.82:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T17:29:25.286 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:29:07.966095+0000 mgr.y (mgr.14152) 45 : audit [DBG] from='client.14184 v1:172.21.15.82:0/3357327228' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi032:[v1:172.21.15.32:6789]=a;smithi032:[v1:172.21.15.32:6790]=c;smithi082:[v1:172.21.15.82:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T17:29:25.286 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cephadm 2024-04-24T17:29:07.972279+0000 mgr.y (mgr.14152) 46 : cephadm [INF] Saving service mon spec with placement smithi032:[v1:172.21.15.32:6789]=a;smithi032:[v1:172.21.15.32:6790]=c;smithi082:[v1:172.21.15.82:6789]=b;count:3 2024-04-24T17:29:25.286 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cephadm 2024-04-24T17:29:07.972279+0000 mgr.y (mgr.14152) 46 : cephadm [INF] Saving service mon spec with placement smithi032:[v1:172.21.15.32:6789]=a;smithi032:[v1:172.21.15.32:6790]=c;smithi082:[v1:172.21.15.82:6789]=b;count:3 2024-04-24T17:29:25.286 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:29:07.977751+0000 mon.a (mon.0) 130 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:25.286 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:29:07.977751+0000 mon.a (mon.0) 130 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:25.286 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:29:07.991758+0000 mon.a (mon.0) 131 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T17:29:25.286 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:29:07.991758+0000 mon.a (mon.0) 131 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T17:29:25.286 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:29:08.835860+0000 mgr.y (mgr.14152) 47 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.286 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:29:08.835860+0000 mgr.y (mgr.14152) 47 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.286 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:29:11.094590+0000 mon.a (mon.0) 132 : audit [DBG] from='client.? v1:172.21.15.82:0/1969215094' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T17:29:25.286 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:29:11.094590+0000 mon.a (mon.0) 132 : audit [DBG] from='client.? v1:172.21.15.82:0/1969215094' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T17:29:25.286 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:29:10.836794+0000 mgr.y (mgr.14152) 48 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.286 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:29:10.836794+0000 mgr.y (mgr.14152) 48 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.286 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:29:12.837807+0000 mgr.y (mgr.14152) 49 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.287 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:29:12.837807+0000 mgr.y (mgr.14152) 49 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.287 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:29:15.028832+0000 mon.a (mon.0) 133 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:25.287 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:29:15.028832+0000 mon.a (mon.0) 133 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:25.287 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:29:15.030708+0000 mon.a (mon.0) 134 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T17:29:25.287 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:29:15.030708+0000 mon.a (mon.0) 134 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T17:29:25.287 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:29:15.040843+0000 mon.a (mon.0) 135 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:25.287 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:29:15.040843+0000 mon.a (mon.0) 135 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:25.287 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:29:15.045051+0000 mon.a (mon.0) 136 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T17:29:25.287 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:29:15.045051+0000 mon.a (mon.0) 136 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T17:29:25.287 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:29:15.046772+0000 mon.a (mon.0) 137 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:25.287 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:29:15.046772+0000 mon.a (mon.0) 137 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:25.287 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:29:14.838776+0000 mgr.y (mgr.14152) 50 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.287 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:29:14.838776+0000 mgr.y (mgr.14152) 50 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.287 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:29:15.033644+0000 mgr.y (mgr.14152) 51 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.287 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:29:15.033644+0000 mgr.y (mgr.14152) 51 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.288 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:29:15.034964+0000 mgr.y (mgr.14152) 52 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.288 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:29:15.034964+0000 mgr.y (mgr.14152) 52 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.288 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cephadm 2024-04-24T17:29:15.048701+0000 mgr.y (mgr.14152) 53 : cephadm [INF] Deploying daemon mon.b on smithi082 2024-04-24T17:29:25.288 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cephadm 2024-04-24T17:29:15.048701+0000 mgr.y (mgr.14152) 53 : cephadm [INF] Deploying daemon mon.b on smithi082 2024-04-24T17:29:25.288 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:29:15.155067+0000 mon.a (mon.0) 138 : audit [DBG] from='client.? v1:172.21.15.82:0/2880720608' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T17:29:25.288 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:29:15.155067+0000 mon.a (mon.0) 138 : audit [DBG] from='client.? v1:172.21.15.82:0/2880720608' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T17:29:25.288 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:29:17.035890+0000 mgr.y (mgr.14152) 54 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.288 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:29:17.035890+0000 mgr.y (mgr.14152) 54 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.288 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:29:19.036804+0000 mgr.y (mgr.14152) 55 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.288 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cluster 2024-04-24T17:29:19.036804+0000 mgr.y (mgr.14152) 55 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:25.288 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:29:19.735876+0000 mon.a (mon.0) 139 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:25.288 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:29:19.735876+0000 mon.a (mon.0) 139 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:25.288 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:29:19.781566+0000 mon.a (mon.0) 140 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:25.288 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:29:19.781566+0000 mon.a (mon.0) 140 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:25.288 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:29:19.783029+0000 mon.a (mon.0) 141 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T17:29:25.289 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:29:19.783029+0000 mon.a (mon.0) 141 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T17:29:25.289 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:29:19.784673+0000 mon.a (mon.0) 142 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:25.289 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:29:19.784673+0000 mon.a (mon.0) 142 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:25.289 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cephadm 2024-04-24T17:29:19.786666+0000 mgr.y (mgr.14152) 56 : cephadm [INF] Deploying daemon mon.c on smithi032 2024-04-24T17:29:25.289 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: cephadm 2024-04-24T17:29:19.786666+0000 mgr.y (mgr.14152) 56 : cephadm [INF] Deploying daemon mon.c on smithi032 2024-04-24T17:29:25.289 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:29:20.164933+0000 mon.a (mon.0) 143 : audit [DBG] from='client.? v1:172.21.15.82:0/3086341464' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T17:29:25.289 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: audit 2024-04-24T17:29:20.164933+0000 mon.a (mon.0) 143 : audit [DBG] from='client.? v1:172.21.15.82:0/3086341464' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T17:29:25.289 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.982+0000 7f167b2f8700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1713979764985402, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 14, "file_size": 2145826, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 6, "largest_seqno": 1144, "table_properties": {"data_size": 2139980, "index_size": 2598, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 2181, "raw_key_size": 25631, "raw_average_key_size": 30, "raw_value_size": 2122393, "raw_average_value_size": 2508, "num_data_blocks": 123, "num_entries": 846, "num_filter_entries": 846, "num_deletions": 20, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713979764, "oldest_key_time": 1713979764, "file_creation_time": 1713979764, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "8bea2976-0059-4b87-a837-82741a50e643", "db_session_id": "ALRHCRHOH7RCAYFGITWW", "orig_file_number": 14, "seqno_to_time_mapping": "N/A"}} 2024-04-24T17:29:25.289 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.982+0000 7f167b2f8700 4 rocksdb: [db/flush_job.cc:1019] [default] [JOB 3] Flush lasted 6106 microseconds, and 4969 cpu microseconds. 2024-04-24T17:29:25.289 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.982+0000 7f167b2f8700 4 rocksdb: (Original Log Time 2024/04/24-17:29:24.985439) [db/flush_job.cc:967] [default] [JOB 3] Level-0 flush table #14: 2145826 bytes OK 2024-04-24T17:29:25.289 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.982+0000 7f167b2f8700 4 rocksdb: (Original Log Time 2024/04/24-17:29:24.985459) [db/memtable_list.cc:519] [default] Level-0 commit table #14 started 2024-04-24T17:29:25.289 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.982+0000 7f167b2f8700 4 rocksdb: (Original Log Time 2024/04/24-17:29:24.985612) [db/memtable_list.cc:722] [default] Level-0 commit table #14: memtable #1 done 2024-04-24T17:29:25.289 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.982+0000 7f167b2f8700 4 rocksdb: (Original Log Time 2024/04/24-17:29:24.985632) EVENT_LOG_v1 {"time_micros": 1713979764985626, "job": 3, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [2, 0, 0, 0, 0, 0, 0], "immutable_memtables": 0} 2024-04-24T17:29:25.289 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.982+0000 7f167b2f8700 4 rocksdb: (Original Log Time 2024/04/24-17:29:24.985648) [db/db_impl/db_impl_compaction_flush.cc:299] [default] Level summary: files[2 0 0 0 0 0 0] max score 0.50 2024-04-24T17:29:25.290 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.982+0000 7f167b2f8700 4 rocksdb: [db/db_impl/db_impl_files.cc:463] [JOB 3] Try to delete WAL files size 2552107, prev total WAL file size 2552330, number of live WAL files 2. 2024-04-24T17:29:25.290 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.982+0000 7f1676aef700 1 mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-24T17:29:25.290 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.982+0000 7f167b2f8700 4 rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-24T17:29:25.290 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.982+0000 7f167baf9700 4 rocksdb: (Original Log Time 2024/04/24-17:29:24.986189) [db/db_impl/db_impl_compaction_flush.cc:3165] [default] Manual compaction from level-0 to level-6 from '6D6772737461740030' seq:72057594037927935, type:22 .. '6D677273746174003231' seq:0, type:0; will stop at (end) 2024-04-24T17:29:25.290 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.982+0000 7f167baf9700 4 rocksdb: [db/compaction/compaction_job.cc:1995] [default] [JOB 4] Compacting 2@0 files to L6, score -1.00 2024-04-24T17:29:25.290 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.982+0000 7f167baf9700 4 rocksdb: [db/compaction/compaction_job.cc:2001] [default]: Compaction start summary: Base version 3 Base level 0, inputs: [14(2095KB) 8(1613B)] 2024-04-24T17:29:25.290 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:24 smithi032 bash[27083]: debug 2024-04-24T17:29:24.982+0000 7f167baf9700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1713979764986259, "job": 4, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [14, 8], "score": -1, "input_data_size": 2147439, "oldest_snapshot_seqno": -1} 2024-04-24T17:29:25.715 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/mon.b/config 2024-04-24T17:29:26.521 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cephadm 2024-04-24T17:29:19.786666+0000 mgr.y (mgr.14152) 56 : cephadm [INF] Deploying daemon mon.c on smithi032 2024-04-24T17:29:26.521 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cephadm 2024-04-24T17:29:19.786666+0000 mgr.y (mgr.14152) 56 : cephadm [INF] Deploying daemon mon.c on smithi032 2024-04-24T17:29:26.521 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: audit 2024-04-24T17:29:20.164933+0000 mon.a (mon.0) 143 : audit [DBG] from='client.? v1:172.21.15.82:0/3086341464' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T17:29:26.521 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: audit 2024-04-24T17:29:20.164933+0000 mon.a (mon.0) 143 : audit [DBG] from='client.? v1:172.21.15.82:0/3086341464' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T17:29:26.521 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:21.037808+0000 mgr.y (mgr.14152) 57 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:26.521 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:21.037808+0000 mgr.y (mgr.14152) 57 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:26.522 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: audit 2024-04-24T17:29:21.178426+0000 mon.a (mon.0) 145 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T17:29:26.522 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: audit 2024-04-24T17:29:21.178426+0000 mon.a (mon.0) 145 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T17:29:26.522 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:21.178586+0000 mon.a (mon.0) 146 : cluster [INF] mon.a calling monitor election 2024-04-24T17:29:26.522 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:21.178586+0000 mon.a (mon.0) 146 : cluster [INF] mon.a calling monitor election 2024-04-24T17:29:26.522 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: audit 2024-04-24T17:29:21.199736+0000 mon.a (mon.0) 147 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T17:29:26.522 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: audit 2024-04-24T17:29:21.199736+0000 mon.a (mon.0) 147 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T17:29:26.522 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: audit 2024-04-24T17:29:22.174508+0000 mon.a (mon.0) 148 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T17:29:26.522 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: audit 2024-04-24T17:29:22.174508+0000 mon.a (mon.0) 148 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T17:29:26.522 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:23.038793+0000 mgr.y (mgr.14152) 58 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:26.522 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:23.038793+0000 mgr.y (mgr.14152) 58 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:26.522 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: audit 2024-04-24T17:29:23.174606+0000 mon.a (mon.0) 149 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T17:29:26.522 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: audit 2024-04-24T17:29:23.174606+0000 mon.a (mon.0) 149 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T17:29:26.523 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:23.178079+0000 mon.b (mon.1) 1 : cluster [INF] mon.b calling monitor election 2024-04-24T17:29:26.523 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:23.178079+0000 mon.b (mon.1) 1 : cluster [INF] mon.b calling monitor election 2024-04-24T17:29:26.523 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: audit 2024-04-24T17:29:24.174708+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T17:29:26.523 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: audit 2024-04-24T17:29:24.174708+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T17:29:26.523 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: audit 2024-04-24T17:29:24.989729+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:26.523 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: audit 2024-04-24T17:29:24.989729+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:26.523 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:25.039437+0000 mgr.y (mgr.14152) 59 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:26.523 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:25.039437+0000 mgr.y (mgr.14152) 59 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:26.523 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: audit 2024-04-24T17:29:25.175022+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T17:29:26.523 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: audit 2024-04-24T17:29:25.175022+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T17:29:26.523 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: audit 2024-04-24T17:29:25.989853+0000 mon.a (mon.0) 153 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:26.523 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: audit 2024-04-24T17:29:25.989853+0000 mon.a (mon.0) 153 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:26.523 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: audit 2024-04-24T17:29:26.175159+0000 mon.a (mon.0) 154 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T17:29:26.523 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: audit 2024-04-24T17:29:26.175159+0000 mon.a (mon.0) 154 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T17:29:26.524 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.222801+0000 mon.a (mon.0) 155 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-04-24T17:29:26.524 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.222801+0000 mon.a (mon.0) 155 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-04-24T17:29:26.524 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.228764+0000 mon.a (mon.0) 156 : cluster [DBG] monmap epoch 2 2024-04-24T17:29:26.524 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.228764+0000 mon.a (mon.0) 156 : cluster [DBG] monmap epoch 2 2024-04-24T17:29:26.524 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.228825+0000 mon.a (mon.0) 157 : cluster [DBG] fsid a7981b84-025f-11ef-bc93-c7b262605968 2024-04-24T17:29:26.524 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.228825+0000 mon.a (mon.0) 157 : cluster [DBG] fsid a7981b84-025f-11ef-bc93-c7b262605968 2024-04-24T17:29:26.524 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.228843+0000 mon.a (mon.0) 158 : cluster [DBG] last_changed 2024-04-24T17:29:21.175737+0000 2024-04-24T17:29:26.524 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.228843+0000 mon.a (mon.0) 158 : cluster [DBG] last_changed 2024-04-24T17:29:21.175737+0000 2024-04-24T17:29:26.524 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.228865+0000 mon.a (mon.0) 159 : cluster [DBG] created 2024-04-24T17:26:59.605652+0000 2024-04-24T17:29:26.524 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.228865+0000 mon.a (mon.0) 159 : cluster [DBG] created 2024-04-24T17:26:59.605652+0000 2024-04-24T17:29:26.524 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.228888+0000 mon.a (mon.0) 160 : cluster [DBG] min_mon_release 19 (squid) 2024-04-24T17:29:26.524 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.228888+0000 mon.a (mon.0) 160 : cluster [DBG] min_mon_release 19 (squid) 2024-04-24T17:29:26.524 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.228908+0000 mon.a (mon.0) 161 : cluster [DBG] election_strategy: 1 2024-04-24T17:29:26.524 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.228908+0000 mon.a (mon.0) 161 : cluster [DBG] election_strategy: 1 2024-04-24T17:29:26.524 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.228927+0000 mon.a (mon.0) 162 : cluster [DBG] 0: v1:172.21.15.32:6789/0 mon.a 2024-04-24T17:29:26.524 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.228927+0000 mon.a (mon.0) 162 : cluster [DBG] 0: v1:172.21.15.32:6789/0 mon.a 2024-04-24T17:29:26.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.228945+0000 mon.a (mon.0) 163 : cluster [DBG] 1: v1:172.21.15.82:6789/0 mon.b 2024-04-24T17:29:26.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.228945+0000 mon.a (mon.0) 163 : cluster [DBG] 1: v1:172.21.15.82:6789/0 mon.b 2024-04-24T17:29:26.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.247828+0000 mon.a (mon.0) 164 : cluster [DBG] fsmap 2024-04-24T17:29:26.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.247828+0000 mon.a (mon.0) 164 : cluster [DBG] fsmap 2024-04-24T17:29:26.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.247881+0000 mon.a (mon.0) 165 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-04-24T17:29:26.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.247881+0000 mon.a (mon.0) 165 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-04-24T17:29:26.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248156+0000 mon.a (mon.0) 166 : cluster [DBG] mgrmap e13: y(active, since 72s) 2024-04-24T17:29:26.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248156+0000 mon.a (mon.0) 166 : cluster [DBG] mgrmap e13: y(active, since 72s) 2024-04-24T17:29:26.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248446+0000 mon.a (mon.0) 167 : cluster [WRN] Health detail: HEALTH_WARN failed to probe daemons or devices 2024-04-24T17:29:26.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248446+0000 mon.a (mon.0) 167 : cluster [WRN] Health detail: HEALTH_WARN failed to probe daemons or devices 2024-04-24T17:29:26.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248480+0000 mon.a (mon.0) 168 : cluster [WRN] [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-24T17:29:26.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248480+0000 mon.a (mon.0) 168 : cluster [WRN] [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-24T17:29:26.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248496+0000 mon.a (mon.0) 169 : cluster [WRN] host smithi082 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T17:29:26.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248496+0000 mon.a (mon.0) 169 : cluster [WRN] host smithi082 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T17:29:26.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248513+0000 mon.a (mon.0) 170 : cluster [WRN] Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi082 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpbyjftb4d:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch 2024-04-24T17:29:26.525 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248513+0000 mon.a (mon.0) 170 : cluster [WRN] Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi082 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpbyjftb4d:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch 2024-04-24T17:29:26.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248528+0000 mon.a (mon.0) 171 : cluster [WRN] /usr/bin/docker: stderr docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:26.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248528+0000 mon.a (mon.0) 171 : cluster [WRN] /usr/bin/docker: stderr docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:26.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248547+0000 mon.a (mon.0) 172 : cluster [WRN] /usr/bin/docker: stderr See 'docker run --help'. 2024-04-24T17:29:26.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248547+0000 mon.a (mon.0) 172 : cluster [WRN] /usr/bin/docker: stderr See 'docker run --help'. 2024-04-24T17:29:26.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248565+0000 mon.a (mon.0) 173 : cluster [WRN] Traceback (most recent call last): 2024-04-24T17:29:26.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248565+0000 mon.a (mon.0) 173 : cluster [WRN] Traceback (most recent call last): 2024-04-24T17:29:26.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248587+0000 mon.a (mon.0) 174 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main 2024-04-24T17:29:26.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248587+0000 mon.a (mon.0) 174 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main 2024-04-24T17:29:26.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248606+0000 mon.a (mon.0) 175 : cluster [WRN] return _run_code(code, main_globals, None, 2024-04-24T17:29:26.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248606+0000 mon.a (mon.0) 175 : cluster [WRN] return _run_code(code, main_globals, None, 2024-04-24T17:29:26.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248623+0000 mon.a (mon.0) 176 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 86, in _run_code 2024-04-24T17:29:26.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248623+0000 mon.a (mon.0) 176 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 86, in _run_code 2024-04-24T17:29:26.526 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248643+0000 mon.a (mon.0) 177 : cluster [WRN] exec(code, run_globals) 2024-04-24T17:29:26.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248643+0000 mon.a (mon.0) 177 : cluster [WRN] exec(code, run_globals) 2024-04-24T17:29:26.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248661+0000 mon.a (mon.0) 178 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T17:29:26.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248661+0000 mon.a (mon.0) 178 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T17:29:26.527 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248681+0000 mon.a (mon.0) 179 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T17:29:26.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248681+0000 mon.a (mon.0) 179 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T17:29:26.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248699+0000 mon.a (mon.0) 180 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 408, in _infer_config 2024-04-24T17:29:26.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248699+0000 mon.a (mon.0) 180 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 408, in _infer_config 2024-04-24T17:29:26.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248718+0000 mon.a (mon.0) 181 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 323, in _infer_fsid 2024-04-24T17:29:26.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248718+0000 mon.a (mon.0) 181 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 323, in _infer_fsid 2024-04-24T17:29:26.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248738+0000 mon.a (mon.0) 182 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 436, in _infer_image 2024-04-24T17:29:26.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248738+0000 mon.a (mon.0) 182 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 436, in _infer_image 2024-04-24T17:29:26.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248757+0000 mon.a (mon.0) 183 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 310, in _validate_fsid 2024-04-24T17:29:26.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248757+0000 mon.a (mon.0) 183 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 310, in _validate_fsid 2024-04-24T17:29:26.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248780+0000 mon.a (mon.0) 184 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3380, in command_ceph_volume 2024-04-24T17:29:26.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248780+0000 mon.a (mon.0) 184 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3380, in command_ceph_volume 2024-04-24T17:29:26.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248803+0000 mon.a (mon.0) 185 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T17:29:26.528 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248803+0000 mon.a (mon.0) 185 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T17:29:26.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248821+0000 mon.a (mon.0) 186 : cluster [WRN] RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi082 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpbyjftb4d:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch: docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:26.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248821+0000 mon.a (mon.0) 186 : cluster [WRN] RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi082 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpbyjftb4d:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch: docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:26.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248846+0000 mon.a (mon.0) 187 : cluster [WRN] See 'docker run --help'. 2024-04-24T17:29:26.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248846+0000 mon.a (mon.0) 187 : cluster [WRN] See 'docker run --help'. 2024-04-24T17:29:26.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248865+0000 mon.a (mon.0) 188 : cluster [WRN] host smithi032 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T17:29:26.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248865+0000 mon.a (mon.0) 188 : cluster [WRN] host smithi032 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T17:29:26.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248883+0000 mon.a (mon.0) 189 : cluster [WRN] Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi032 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpo3z7jlle:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch 2024-04-24T17:29:26.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248883+0000 mon.a (mon.0) 189 : cluster [WRN] Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi032 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpo3z7jlle:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch 2024-04-24T17:29:26.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248908+0000 mon.a (mon.0) 190 : cluster [WRN] /usr/bin/docker: stderr docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:26.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248908+0000 mon.a (mon.0) 190 : cluster [WRN] /usr/bin/docker: stderr docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:26.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248926+0000 mon.a (mon.0) 191 : cluster [WRN] /usr/bin/docker: stderr See 'docker run --help'. 2024-04-24T17:29:26.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248926+0000 mon.a (mon.0) 191 : cluster [WRN] /usr/bin/docker: stderr See 'docker run --help'. 2024-04-24T17:29:26.529 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248943+0000 mon.a (mon.0) 192 : cluster [WRN] Traceback (most recent call last): 2024-04-24T17:29:26.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248943+0000 mon.a (mon.0) 192 : cluster [WRN] Traceback (most recent call last): 2024-04-24T17:29:26.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248961+0000 mon.a (mon.0) 193 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main 2024-04-24T17:29:26.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248961+0000 mon.a (mon.0) 193 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main 2024-04-24T17:29:26.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248982+0000 mon.a (mon.0) 194 : cluster [WRN] return _run_code(code, main_globals, None, 2024-04-24T17:29:26.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.248982+0000 mon.a (mon.0) 194 : cluster [WRN] return _run_code(code, main_globals, None, 2024-04-24T17:29:26.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.249000+0000 mon.a (mon.0) 195 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 86, in _run_code 2024-04-24T17:29:26.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.249000+0000 mon.a (mon.0) 195 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 86, in _run_code 2024-04-24T17:29:26.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.249019+0000 mon.a (mon.0) 196 : cluster [WRN] exec(code, run_globals) 2024-04-24T17:29:26.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.249019+0000 mon.a (mon.0) 196 : cluster [WRN] exec(code, run_globals) 2024-04-24T17:29:26.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.249037+0000 mon.a (mon.0) 197 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T17:29:26.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.249037+0000 mon.a (mon.0) 197 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T17:29:26.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.249055+0000 mon.a (mon.0) 198 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T17:29:26.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.249055+0000 mon.a (mon.0) 198 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T17:29:26.530 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.249074+0000 mon.a (mon.0) 199 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 408, in _infer_config 2024-04-24T17:29:26.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.249074+0000 mon.a (mon.0) 199 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 408, in _infer_config 2024-04-24T17:29:26.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.249099+0000 mon.a (mon.0) 200 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 323, in _infer_fsid 2024-04-24T17:29:26.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.249099+0000 mon.a (mon.0) 200 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 323, in _infer_fsid 2024-04-24T17:29:26.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.249118+0000 mon.a (mon.0) 201 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 436, in _infer_image 2024-04-24T17:29:26.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.249118+0000 mon.a (mon.0) 201 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 436, in _infer_image 2024-04-24T17:29:26.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.249139+0000 mon.a (mon.0) 202 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 310, in _validate_fsid 2024-04-24T17:29:26.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.249139+0000 mon.a (mon.0) 202 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 310, in _validate_fsid 2024-04-24T17:29:26.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.249158+0000 mon.a (mon.0) 203 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3380, in command_ceph_volume 2024-04-24T17:29:26.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.249158+0000 mon.a (mon.0) 203 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3380, in command_ceph_volume 2024-04-24T17:29:26.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.249178+0000 mon.a (mon.0) 204 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T17:29:26.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.249178+0000 mon.a (mon.0) 204 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T17:29:26.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.249199+0000 mon.a (mon.0) 205 : cluster [WRN] RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi032 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpo3z7jlle:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch: docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:26.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.249199+0000 mon.a (mon.0) 205 : cluster [WRN] RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi032 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpo3z7jlle:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch: docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:26.531 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.249227+0000 mon.a (mon.0) 206 : cluster [WRN] See 'docker run --help'. 2024-04-24T17:29:26.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.249227+0000 mon.a (mon.0) 206 : cluster [WRN] See 'docker run --help'. 2024-04-24T17:29:26.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: audit 2024-04-24T17:29:26.256924+0000 mon.a (mon.0) 207 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:26.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: audit 2024-04-24T17:29:26.256924+0000 mon.a (mon.0) 207 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:26.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: audit 2024-04-24T17:29:26.272804+0000 mon.a (mon.0) 208 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:26.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: audit 2024-04-24T17:29:26.272804+0000 mon.a (mon.0) 208 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:26.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: audit 2024-04-24T17:29:26.280157+0000 mon.a (mon.0) 209 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:26.532 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:26 smithi082 bash[21793]: audit 2024-04-24T17:29:26.280157+0000 mon.a (mon.0) 209 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:26.690 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:26.685+0000 7f6f2db6b700 1 -- 172.21.15.82:0/3073560163 >> v1:172.21.15.32:6789/0 conn(0x7f6f28102000 legacy=0x7f6f281023e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:29:26.690 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:26.685+0000 7f6f2db6b700 1 -- 172.21.15.82:0/3073560163 shutdown_connections 2024-04-24T17:29:26.690 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:26.685+0000 7f6f2db6b700 1 -- 172.21.15.82:0/3073560163 >> 172.21.15.82:0/3073560163 conn(0x7f6f280fd8b0 msgr2=0x7f6f280ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:29:26.690 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:26.685+0000 7f6f2db6b700 1 -- 172.21.15.82:0/3073560163 shutdown_connections 2024-04-24T17:29:26.690 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:26.685+0000 7f6f2db6b700 1 -- 172.21.15.82:0/3073560163 wait complete. 2024-04-24T17:29:26.691 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:26.685+0000 7f6f2db6b700 1 librados: starting msgr at 2024-04-24T17:29:26.691 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:26.685+0000 7f6f2db6b700 1 librados: starting objecter 2024-04-24T17:29:26.691 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:26.685+0000 7f6f2db6b700 1 Processor -- start 2024-04-24T17:29:26.691 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:26.685+0000 7f6f2db6b700 1 -- start start 2024-04-24T17:29:26.691 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:26.685+0000 7f6f2db6b700 1 librados: setting wanted keys 2024-04-24T17:29:26.691 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:26.685+0000 7f6f2db6b700 1 librados: calling monclient init 2024-04-24T17:29:26.692 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:26.685+0000 7f6f2db6b700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6f281a1d40 con 0x7f6f2819be70 2024-04-24T17:29:26.692 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:26.685+0000 7f6f2db6b700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6f281a2f20 con 0x7f6f28102000 2024-04-24T17:29:26.692 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:26.685+0000 7f6f277fe700 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f6f28102000 0x7f6f2819b760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.82:51384/0 (socket says 172.21.15.82:51384) 2024-04-24T17:29:26.692 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:26.685+0000 7f6f277fe700 1 -- 172.21.15.82:0/2538690701 learned_addr learned my addr 172.21.15.82:0/2538690701 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-04-24T17:29:26.692 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:26.685+0000 7f6f24ff9700 10 client.?.objecter ms_handle_connect 0x7f6f28102000 2024-04-24T17:29:26.692 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:26.685+0000 7f6f24ff9700 10 client.?.objecter resend_mon_ops 2024-04-24T17:29:26.692 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:26.689+0000 7f6f24ff9700 10 client.?.objecter ms_handle_connect 0x7f6f2819be70 2024-04-24T17:29:26.692 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:26.689+0000 7f6f24ff9700 10 client.?.objecter resend_mon_ops 2024-04-24T17:29:26.692 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:26.689+0000 7f6f24ff9700 1 -- 172.21.15.82:0/2538690701 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2439052794 0 0) 0x7f6f281a1d40 con 0x7f6f2819be70 2024-04-24T17:29:26.692 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:26.689+0000 7f6f24ff9700 1 -- 172.21.15.82:0/2538690701 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f100035b0 con 0x7f6f2819be70 2024-04-24T17:29:26.693 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:26.689+0000 7f6f24ff9700 1 -- 172.21.15.82:0/2538690701 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2948126997 0 0) 0x7f6f100035b0 con 0x7f6f2819be70 2024-04-24T17:29:26.693 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:26.689+0000 7f6f24ff9700 1 -- 172.21.15.82:0/2538690701 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6f281a1d40 con 0x7f6f2819be70 2024-04-24T17:29:26.693 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:26.689+0000 7f6f24ff9700 1 -- 172.21.15.82:0/2538690701 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 239+0+0 (unknown 3454299706 0 0) 0x7f6f1c0017e0 con 0x7f6f2819be70 2024-04-24T17:29:26.693 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:26.689+0000 7f6f24ff9700 1 -- 172.21.15.82:0/2538690701 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1449376735 0 0) 0x7f6f281a1d40 con 0x7f6f2819be70 2024-04-24T17:29:26.693 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:26.689+0000 7f6f24ff9700 1 -- 172.21.15.82:0/2538690701 >> v1:172.21.15.82:6789/0 conn(0x7f6f28102000 legacy=0x7f6f2819b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:29:26.694 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:26.689+0000 7f6f24ff9700 1 -- 172.21.15.82:0/2538690701 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f281a4100 con 0x7f6f2819be70 2024-04-24T17:29:26.694 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:26.689+0000 7f6f2db6b700 1 -- 172.21.15.82:0/2538690701 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6f281a1f70 con 0x7f6f2819be70 2024-04-24T17:29:26.694 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:26.689+0000 7f6f2db6b700 10 client.14194.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-24T17:29:26.694 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:26.689+0000 7f6f2db6b700 1 -- 172.21.15.82:0/2538690701 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6f281a2470 con 0x7f6f2819be70 2024-04-24T17:29:26.694 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:26.689+0000 7f6f2db6b700 1 librados: init done 2024-04-24T17:29:26.694 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:26.689+0000 7f6f2db6b700 1 -- 172.21.15.82:0/2538690701 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f28064a80 con 0x7f6f2819be70 2024-04-24T17:29:26.697 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:26.693+0000 7f6f24ff9700 1 -- 172.21.15.82:0/2538690701 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(25 keys) ==== 1042+0+0 (unknown 2768897130 0 0) 0x7f6f1c001c70 con 0x7f6f2819be70 2024-04-24T17:29:26.697 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:26.693+0000 7f6f24ff9700 1 -- 172.21.15.82:0/2538690701 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 ==== 239+0+0 (unknown 3454299706 0 0) 0x7f6f1c004a60 con 0x7f6f2819be70 2024-04-24T17:29:26.698 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:26.693+0000 7f6f24ff9700 1 -- 172.21.15.82:0/2538690701 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 13) ==== 46579+0+0 (unknown 583022566 0 0) 0x7f6f1c010200 con 0x7f6f2819be70 2024-04-24T17:29:26.698 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:26.693+0000 7f6f24ff9700 1 -- 172.21.15.82:0/2538690701 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(4..4 src has 1..4) ==== 1069+0+0 (unknown 1280957937 0 0) 0x7f6f1c046e30 con 0x7f6f2819be70 2024-04-24T17:29:26.699 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:26.693+0000 7f6f24ff9700 10 client.14194.objecter ms_dispatch 0x7f6f28001550 osd_map(4..4 src has 1..4) 2024-04-24T17:29:26.699 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:26.693+0000 7f6f24ff9700 3 client.14194.objecter handle_osd_map got epochs [4,4] > 0 2024-04-24T17:29:26.699 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:26.693+0000 7f6f24ff9700 3 client.14194.objecter handle_osd_map decoding full epoch 4 2024-04-24T17:29:26.699 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:26.693+0000 7f6f24ff9700 20 client.14194.objecter dump_active .. 0 homeless 2024-04-24T17:29:26.699 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:26.693+0000 7f6f24ff9700 10 client.14194.objecter ms_handle_connect 0x7f6f1003e650 2024-04-24T17:29:26.712 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:26.709+0000 7f6f24ff9700 1 -- 172.21.15.82:0/2538690701 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188728 (unknown 1092875540 0 2656817166) 0x7f6f1c016170 con 0x7f6f2819be70 2024-04-24T17:29:26.752 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:21.037808+0000 mgr.y (mgr.14152) 57 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:26.753 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:21.037808+0000 mgr.y (mgr.14152) 57 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:26.753 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: audit 2024-04-24T17:29:21.178426+0000 mon.a (mon.0) 145 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T17:29:26.753 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: audit 2024-04-24T17:29:21.178426+0000 mon.a (mon.0) 145 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T17:29:26.753 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:21.178586+0000 mon.a (mon.0) 146 : cluster [INF] mon.a calling monitor election 2024-04-24T17:29:26.753 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:21.178586+0000 mon.a (mon.0) 146 : cluster [INF] mon.a calling monitor election 2024-04-24T17:29:26.753 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: audit 2024-04-24T17:29:21.199736+0000 mon.a (mon.0) 147 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T17:29:26.753 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: audit 2024-04-24T17:29:21.199736+0000 mon.a (mon.0) 147 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T17:29:26.753 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: audit 2024-04-24T17:29:22.174508+0000 mon.a (mon.0) 148 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T17:29:26.753 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: audit 2024-04-24T17:29:22.174508+0000 mon.a (mon.0) 148 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T17:29:26.753 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:23.038793+0000 mgr.y (mgr.14152) 58 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:26.753 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:23.038793+0000 mgr.y (mgr.14152) 58 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:26.753 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: audit 2024-04-24T17:29:23.174606+0000 mon.a (mon.0) 149 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T17:29:26.753 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: audit 2024-04-24T17:29:23.174606+0000 mon.a (mon.0) 149 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T17:29:26.754 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:23.178079+0000 mon.b (mon.1) 1 : cluster [INF] mon.b calling monitor election 2024-04-24T17:29:26.754 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:23.178079+0000 mon.b (mon.1) 1 : cluster [INF] mon.b calling monitor election 2024-04-24T17:29:26.754 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: audit 2024-04-24T17:29:24.174708+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T17:29:26.754 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: audit 2024-04-24T17:29:24.174708+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T17:29:26.754 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: audit 2024-04-24T17:29:24.989729+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:26.754 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: audit 2024-04-24T17:29:24.989729+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:26.755 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:25.039437+0000 mgr.y (mgr.14152) 59 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:26.755 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:25.039437+0000 mgr.y (mgr.14152) 59 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:26.755 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: audit 2024-04-24T17:29:25.175022+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T17:29:26.755 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: audit 2024-04-24T17:29:25.175022+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T17:29:26.755 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: audit 2024-04-24T17:29:25.989853+0000 mon.a (mon.0) 153 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:26.755 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: audit 2024-04-24T17:29:25.989853+0000 mon.a (mon.0) 153 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:26.755 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: audit 2024-04-24T17:29:26.175159+0000 mon.a (mon.0) 154 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T17:29:26.755 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: audit 2024-04-24T17:29:26.175159+0000 mon.a (mon.0) 154 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T17:29:26.755 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.222801+0000 mon.a (mon.0) 155 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-04-24T17:29:26.755 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.222801+0000 mon.a (mon.0) 155 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-04-24T17:29:26.755 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.228764+0000 mon.a (mon.0) 156 : cluster [DBG] monmap epoch 2 2024-04-24T17:29:26.755 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.228764+0000 mon.a (mon.0) 156 : cluster [DBG] monmap epoch 2 2024-04-24T17:29:26.756 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.228825+0000 mon.a (mon.0) 157 : cluster [DBG] fsid a7981b84-025f-11ef-bc93-c7b262605968 2024-04-24T17:29:26.756 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.228825+0000 mon.a (mon.0) 157 : cluster [DBG] fsid a7981b84-025f-11ef-bc93-c7b262605968 2024-04-24T17:29:26.756 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.228843+0000 mon.a (mon.0) 158 : cluster [DBG] last_changed 2024-04-24T17:29:21.175737+0000 2024-04-24T17:29:26.756 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.228843+0000 mon.a (mon.0) 158 : cluster [DBG] last_changed 2024-04-24T17:29:21.175737+0000 2024-04-24T17:29:26.756 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.228865+0000 mon.a (mon.0) 159 : cluster [DBG] created 2024-04-24T17:26:59.605652+0000 2024-04-24T17:29:26.756 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.228865+0000 mon.a (mon.0) 159 : cluster [DBG] created 2024-04-24T17:26:59.605652+0000 2024-04-24T17:29:26.756 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.228888+0000 mon.a (mon.0) 160 : cluster [DBG] min_mon_release 19 (squid) 2024-04-24T17:29:26.756 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.228888+0000 mon.a (mon.0) 160 : cluster [DBG] min_mon_release 19 (squid) 2024-04-24T17:29:26.756 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.228908+0000 mon.a (mon.0) 161 : cluster [DBG] election_strategy: 1 2024-04-24T17:29:26.756 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.228908+0000 mon.a (mon.0) 161 : cluster [DBG] election_strategy: 1 2024-04-24T17:29:26.756 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.228927+0000 mon.a (mon.0) 162 : cluster [DBG] 0: v1:172.21.15.32:6789/0 mon.a 2024-04-24T17:29:26.756 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.228927+0000 mon.a (mon.0) 162 : cluster [DBG] 0: v1:172.21.15.32:6789/0 mon.a 2024-04-24T17:29:26.756 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.228945+0000 mon.a (mon.0) 163 : cluster [DBG] 1: v1:172.21.15.82:6789/0 mon.b 2024-04-24T17:29:26.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.228945+0000 mon.a (mon.0) 163 : cluster [DBG] 1: v1:172.21.15.82:6789/0 mon.b 2024-04-24T17:29:26.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.247828+0000 mon.a (mon.0) 164 : cluster [DBG] fsmap 2024-04-24T17:29:26.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.247828+0000 mon.a (mon.0) 164 : cluster [DBG] fsmap 2024-04-24T17:29:26.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.247881+0000 mon.a (mon.0) 165 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-04-24T17:29:26.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.247881+0000 mon.a (mon.0) 165 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-04-24T17:29:26.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248156+0000 mon.a (mon.0) 166 : cluster [DBG] mgrmap e13: y(active, since 72s) 2024-04-24T17:29:26.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248156+0000 mon.a (mon.0) 166 : cluster [DBG] mgrmap e13: y(active, since 72s) 2024-04-24T17:29:26.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248446+0000 mon.a (mon.0) 167 : cluster [WRN] Health detail: HEALTH_WARN failed to probe daemons or devices 2024-04-24T17:29:26.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248446+0000 mon.a (mon.0) 167 : cluster [WRN] Health detail: HEALTH_WARN failed to probe daemons or devices 2024-04-24T17:29:26.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248480+0000 mon.a (mon.0) 168 : cluster [WRN] [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-24T17:29:26.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248480+0000 mon.a (mon.0) 168 : cluster [WRN] [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-24T17:29:26.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248496+0000 mon.a (mon.0) 169 : cluster [WRN] host smithi082 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T17:29:26.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248496+0000 mon.a (mon.0) 169 : cluster [WRN] host smithi082 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T17:29:26.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248513+0000 mon.a (mon.0) 170 : cluster [WRN] Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi082 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpbyjftb4d:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch 2024-04-24T17:29:26.757 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248513+0000 mon.a (mon.0) 170 : cluster [WRN] Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi082 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpbyjftb4d:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch 2024-04-24T17:29:26.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248528+0000 mon.a (mon.0) 171 : cluster [WRN] /usr/bin/docker: stderr docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:26.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248528+0000 mon.a (mon.0) 171 : cluster [WRN] /usr/bin/docker: stderr docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:26.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248547+0000 mon.a (mon.0) 172 : cluster [WRN] /usr/bin/docker: stderr See 'docker run --help'. 2024-04-24T17:29:26.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248547+0000 mon.a (mon.0) 172 : cluster [WRN] /usr/bin/docker: stderr See 'docker run --help'. 2024-04-24T17:29:26.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248565+0000 mon.a (mon.0) 173 : cluster [WRN] Traceback (most recent call last): 2024-04-24T17:29:26.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248565+0000 mon.a (mon.0) 173 : cluster [WRN] Traceback (most recent call last): 2024-04-24T17:29:26.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248587+0000 mon.a (mon.0) 174 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main 2024-04-24T17:29:26.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248587+0000 mon.a (mon.0) 174 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main 2024-04-24T17:29:26.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248606+0000 mon.a (mon.0) 175 : cluster [WRN] return _run_code(code, main_globals, None, 2024-04-24T17:29:26.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248606+0000 mon.a (mon.0) 175 : cluster [WRN] return _run_code(code, main_globals, None, 2024-04-24T17:29:26.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248623+0000 mon.a (mon.0) 176 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 86, in _run_code 2024-04-24T17:29:26.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248623+0000 mon.a (mon.0) 176 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 86, in _run_code 2024-04-24T17:29:26.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248643+0000 mon.a (mon.0) 177 : cluster [WRN] exec(code, run_globals) 2024-04-24T17:29:26.758 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248643+0000 mon.a (mon.0) 177 : cluster [WRN] exec(code, run_globals) 2024-04-24T17:29:26.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248661+0000 mon.a (mon.0) 178 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T17:29:26.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248661+0000 mon.a (mon.0) 178 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T17:29:26.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248681+0000 mon.a (mon.0) 179 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T17:29:26.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248681+0000 mon.a (mon.0) 179 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T17:29:26.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248699+0000 mon.a (mon.0) 180 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 408, in _infer_config 2024-04-24T17:29:26.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248699+0000 mon.a (mon.0) 180 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 408, in _infer_config 2024-04-24T17:29:26.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248718+0000 mon.a (mon.0) 181 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 323, in _infer_fsid 2024-04-24T17:29:26.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248718+0000 mon.a (mon.0) 181 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 323, in _infer_fsid 2024-04-24T17:29:26.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248738+0000 mon.a (mon.0) 182 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 436, in _infer_image 2024-04-24T17:29:26.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248738+0000 mon.a (mon.0) 182 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 436, in _infer_image 2024-04-24T17:29:26.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248757+0000 mon.a (mon.0) 183 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 310, in _validate_fsid 2024-04-24T17:29:26.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248757+0000 mon.a (mon.0) 183 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 310, in _validate_fsid 2024-04-24T17:29:26.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248780+0000 mon.a (mon.0) 184 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3380, in command_ceph_volume 2024-04-24T17:29:26.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248780+0000 mon.a (mon.0) 184 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3380, in command_ceph_volume 2024-04-24T17:29:26.759 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248803+0000 mon.a (mon.0) 185 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T17:29:26.760 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248803+0000 mon.a (mon.0) 185 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T17:29:26.760 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248821+0000 mon.a (mon.0) 186 : cluster [WRN] RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi082 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpbyjftb4d:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch: docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:26.760 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248821+0000 mon.a (mon.0) 186 : cluster [WRN] RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi082 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpbyjftb4d:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch: docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:26.760 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248846+0000 mon.a (mon.0) 187 : cluster [WRN] See 'docker run --help'. 2024-04-24T17:29:26.760 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248846+0000 mon.a (mon.0) 187 : cluster [WRN] See 'docker run --help'. 2024-04-24T17:29:26.760 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248865+0000 mon.a (mon.0) 188 : cluster [WRN] host smithi032 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T17:29:26.761 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248865+0000 mon.a (mon.0) 188 : cluster [WRN] host smithi032 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T17:29:26.761 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248883+0000 mon.a (mon.0) 189 : cluster [WRN] Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi032 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpo3z7jlle:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch 2024-04-24T17:29:26.761 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248883+0000 mon.a (mon.0) 189 : cluster [WRN] Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi032 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpo3z7jlle:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch 2024-04-24T17:29:26.761 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248908+0000 mon.a (mon.0) 190 : cluster [WRN] /usr/bin/docker: stderr docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:26.761 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248908+0000 mon.a (mon.0) 190 : cluster [WRN] /usr/bin/docker: stderr docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:26.761 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248926+0000 mon.a (mon.0) 191 : cluster [WRN] /usr/bin/docker: stderr See 'docker run --help'. 2024-04-24T17:29:26.761 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248926+0000 mon.a (mon.0) 191 : cluster [WRN] /usr/bin/docker: stderr See 'docker run --help'. 2024-04-24T17:29:26.761 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248943+0000 mon.a (mon.0) 192 : cluster [WRN] Traceback (most recent call last): 2024-04-24T17:29:26.761 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248943+0000 mon.a (mon.0) 192 : cluster [WRN] Traceback (most recent call last): 2024-04-24T17:29:26.761 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248961+0000 mon.a (mon.0) 193 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main 2024-04-24T17:29:26.761 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248961+0000 mon.a (mon.0) 193 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main 2024-04-24T17:29:26.761 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248982+0000 mon.a (mon.0) 194 : cluster [WRN] return _run_code(code, main_globals, None, 2024-04-24T17:29:26.761 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.248982+0000 mon.a (mon.0) 194 : cluster [WRN] return _run_code(code, main_globals, None, 2024-04-24T17:29:26.761 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.249000+0000 mon.a (mon.0) 195 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 86, in _run_code 2024-04-24T17:29:26.761 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.249000+0000 mon.a (mon.0) 195 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 86, in _run_code 2024-04-24T17:29:26.762 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.249019+0000 mon.a (mon.0) 196 : cluster [WRN] exec(code, run_globals) 2024-04-24T17:29:26.762 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.249019+0000 mon.a (mon.0) 196 : cluster [WRN] exec(code, run_globals) 2024-04-24T17:29:26.762 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.249037+0000 mon.a (mon.0) 197 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T17:29:26.762 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.249037+0000 mon.a (mon.0) 197 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T17:29:26.762 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.249055+0000 mon.a (mon.0) 198 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T17:29:26.762 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.249055+0000 mon.a (mon.0) 198 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T17:29:26.762 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.249074+0000 mon.a (mon.0) 199 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 408, in _infer_config 2024-04-24T17:29:26.762 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.249074+0000 mon.a (mon.0) 199 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 408, in _infer_config 2024-04-24T17:29:26.762 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.249099+0000 mon.a (mon.0) 200 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 323, in _infer_fsid 2024-04-24T17:29:26.762 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.249099+0000 mon.a (mon.0) 200 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 323, in _infer_fsid 2024-04-24T17:29:26.762 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.249118+0000 mon.a (mon.0) 201 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 436, in _infer_image 2024-04-24T17:29:26.762 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.249118+0000 mon.a (mon.0) 201 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 436, in _infer_image 2024-04-24T17:29:26.762 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.249139+0000 mon.a (mon.0) 202 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 310, in _validate_fsid 2024-04-24T17:29:26.763 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.249139+0000 mon.a (mon.0) 202 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 310, in _validate_fsid 2024-04-24T17:29:26.763 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.249158+0000 mon.a (mon.0) 203 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3380, in command_ceph_volume 2024-04-24T17:29:26.763 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.249158+0000 mon.a (mon.0) 203 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3380, in command_ceph_volume 2024-04-24T17:29:26.763 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.249178+0000 mon.a (mon.0) 204 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T17:29:26.763 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.249178+0000 mon.a (mon.0) 204 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T17:29:26.763 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.249199+0000 mon.a (mon.0) 205 : cluster [WRN] RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi032 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpo3z7jlle:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch: docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:26.763 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.249199+0000 mon.a (mon.0) 205 : cluster [WRN] RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi032 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpo3z7jlle:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch: docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:26.763 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.249227+0000 mon.a (mon.0) 206 : cluster [WRN] See 'docker run --help'. 2024-04-24T17:29:26.763 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.249227+0000 mon.a (mon.0) 206 : cluster [WRN] See 'docker run --help'. 2024-04-24T17:29:26.763 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: audit 2024-04-24T17:29:26.256924+0000 mon.a (mon.0) 207 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:26.763 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: audit 2024-04-24T17:29:26.256924+0000 mon.a (mon.0) 207 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:26.763 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: audit 2024-04-24T17:29:26.272804+0000 mon.a (mon.0) 208 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:26.763 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: audit 2024-04-24T17:29:26.272804+0000 mon.a (mon.0) 208 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:26.763 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: audit 2024-04-24T17:29:26.280157+0000 mon.a (mon.0) 209 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:26.763 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:26 smithi032 bash[21457]: audit 2024-04-24T17:29:26.280157+0000 mon.a (mon.0) 209 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:26.992 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:26.989+0000 7f6f24ff9700 1 -- 172.21.15.82:0/2538690701 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_map magic: 0 ==== 308+0+0 (unknown 512931633 0 0) 0x7f6f1c013c10 con 0x7f6f2819be70 2024-04-24T17:29:27.005 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:27.001+0000 7f6f2db6b700 1 -- 172.21.15.82:0/2538690701 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f6f2806c220 con 0x7f6f2819be70 2024-04-24T17:29:27.502 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:29:27 smithi032 bash[21712]: debug 2024-04-24T17:29:27.174+0000 7fd3d5282700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-24T17:29:31.696 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:31.693+0000 7f6f2d36a700 10 client.14194.objecter tick 2024-04-24T17:29:32.073 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:32.073+0000 7f6f24ff9700 1 -- 172.21.15.82:0/2538690701 <== mon.0 v1:172.21.15.32:6789/0 11 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1124 (unknown 409899127 0 1546913359) 0x7f6f1c015a70 con 0x7f6f2819be70 2024-04-24T17:29:32.073 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-04-24T17:29:32.073 INFO:teuthology.orchestra.run.smithi082.stdout:{"epoch":3,"fsid":"a7981b84-025f-11ef-bc93-c7b262605968","modified":"2024-04-24T17:29:26.991555Z","created":"2024-04-24T17:26:59.605652Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.32:6790","nonce":0}]},"addr":"172.21.15.32:6790/0","public_addr":"172.21.15.32:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-04-24T17:29:32.076 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:32.073+0000 7f6f2db6b700 10 librados: watch_flush enter 2024-04-24T17:29:32.076 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:32.073+0000 7f6f2db6b700 10 librados: watch_flush exit 2024-04-24T17:29:32.076 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:32.073+0000 7f6f2db6b700 20 client.14194.objecter shutdown clearing up homeless session... 2024-04-24T17:29:32.076 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:32.073+0000 7f6f2db6b700 10 client.14194.objecter successfully canceled tick 2024-04-24T17:29:32.076 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:32.073+0000 7f6f2db6b700 1 -- 172.21.15.82:0/2538690701 >> v1:172.21.15.32:6800/1847389373 conn(0x7f6f1003e650 legacy=0x7f6f10040b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:29:32.077 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:32.073+0000 7f6f2db6b700 1 -- 172.21.15.82:0/2538690701 >> v1:172.21.15.32:6789/0 conn(0x7f6f2819be70 legacy=0x7f6f281a0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:29:32.077 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:32.073+0000 7f6f2db6b700 1 -- 172.21.15.82:0/2538690701 shutdown_connections 2024-04-24T17:29:32.077 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:32.073+0000 7f6f2db6b700 1 -- 172.21.15.82:0/2538690701 >> 172.21.15.82:0/2538690701 conn(0x7f6f280fd8b0 msgr2=0x7f6f28109280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:29:32.077 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:32.073+0000 7f6f2db6b700 1 -- 172.21.15.82:0/2538690701 shutdown_connections 2024-04-24T17:29:32.077 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:32.073+0000 7f6f2db6b700 1 -- 172.21.15.82:0/2538690701 wait complete. 2024-04-24T17:29:32.077 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:32.073+0000 7f6f2db6b700 1 librados: shutdown 2024-04-24T17:29:32.077 INFO:teuthology.orchestra.run.smithi082.stderr:dumped monmap epoch 3 2024-04-24T17:29:32.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: audit 2024-04-24T17:29:26.998218+0000 mon.a (mon.0) 212 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T17:29:32.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: audit 2024-04-24T17:29:26.998218+0000 mon.a (mon.0) 212 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T17:29:32.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.998654+0000 mon.a (mon.0) 213 : cluster [INF] mon.a calling monitor election 2024-04-24T17:29:32.381 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:26.998654+0000 mon.a (mon.0) 213 : cluster [INF] mon.a calling monitor election 2024-04-24T17:29:32.381 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:27.003312+0000 mon.b (mon.1) 2 : cluster [INF] mon.b calling monitor election 2024-04-24T17:29:32.381 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:27.003312+0000 mon.b (mon.1) 2 : cluster [INF] mon.b calling monitor election 2024-04-24T17:29:32.381 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: audit 2024-04-24T17:29:27.020403+0000 mon.a (mon.0) 214 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T17:29:32.381 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: audit 2024-04-24T17:29:27.020403+0000 mon.a (mon.0) 214 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T17:29:32.381 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: audit 2024-04-24T17:29:27.020738+0000 mon.a (mon.0) 215 : audit [DBG] from='client.? v1:172.21.15.82:0/2538690701' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T17:29:32.381 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: audit 2024-04-24T17:29:27.020738+0000 mon.a (mon.0) 215 : audit [DBG] from='client.? v1:172.21.15.82:0/2538690701' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T17:29:32.381 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: audit 2024-04-24T17:29:27.020961+0000 mon.a (mon.0) 216 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:32.381 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: audit 2024-04-24T17:29:27.020961+0000 mon.a (mon.0) 216 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:32.381 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:27.040410+0000 mgr.y (mgr.14152) 60 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:32.381 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:27.040410+0000 mgr.y (mgr.14152) 60 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:32.381 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: audit 2024-04-24T17:29:27.990231+0000 mon.a (mon.0) 217 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:32.381 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: audit 2024-04-24T17:29:27.990231+0000 mon.a (mon.0) 217 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:32.381 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: audit 2024-04-24T17:29:28.990307+0000 mon.a (mon.0) 218 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:32.381 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: audit 2024-04-24T17:29:28.990307+0000 mon.a (mon.0) 218 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:32.381 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:28.993624+0000 mon.c (mon.2) 1 : cluster [INF] mon.c calling monitor election 2024-04-24T17:29:32.382 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:28.993624+0000 mon.c (mon.2) 1 : cluster [INF] mon.c calling monitor election 2024-04-24T17:29:32.382 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:29.041381+0000 mgr.y (mgr.14152) 61 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:32.383 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:29.041381+0000 mgr.y (mgr.14152) 61 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:32.383 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: audit 2024-04-24T17:29:29.990698+0000 mon.a (mon.0) 219 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:32.383 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: audit 2024-04-24T17:29:29.990698+0000 mon.a (mon.0) 219 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:32.383 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: audit 2024-04-24T17:29:30.990765+0000 mon.a (mon.0) 220 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:32.383 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: audit 2024-04-24T17:29:30.990765+0000 mon.a (mon.0) 220 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:32.383 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:31.042310+0000 mgr.y (mgr.14152) 62 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:32.383 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:31.042310+0000 mgr.y (mgr.14152) 62 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:32.383 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: audit 2024-04-24T17:29:31.990982+0000 mon.a (mon.0) 221 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:32.383 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: audit 2024-04-24T17:29:31.990982+0000 mon.a (mon.0) 221 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:32.383 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.042895+0000 mon.a (mon.0) 222 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-04-24T17:29:32.383 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.042895+0000 mon.a (mon.0) 222 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-04-24T17:29:32.383 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.050928+0000 mon.a (mon.0) 223 : cluster [DBG] monmap epoch 3 2024-04-24T17:29:32.383 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.050928+0000 mon.a (mon.0) 223 : cluster [DBG] monmap epoch 3 2024-04-24T17:29:32.383 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.050983+0000 mon.a (mon.0) 224 : cluster [DBG] fsid a7981b84-025f-11ef-bc93-c7b262605968 2024-04-24T17:29:32.384 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.050983+0000 mon.a (mon.0) 224 : cluster [DBG] fsid a7981b84-025f-11ef-bc93-c7b262605968 2024-04-24T17:29:32.384 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.051029+0000 mon.a (mon.0) 225 : cluster [DBG] last_changed 2024-04-24T17:29:26.991555+0000 2024-04-24T17:29:32.384 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.051029+0000 mon.a (mon.0) 225 : cluster [DBG] last_changed 2024-04-24T17:29:26.991555+0000 2024-04-24T17:29:32.384 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.051052+0000 mon.a (mon.0) 226 : cluster [DBG] created 2024-04-24T17:26:59.605652+0000 2024-04-24T17:29:32.384 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.051052+0000 mon.a (mon.0) 226 : cluster [DBG] created 2024-04-24T17:26:59.605652+0000 2024-04-24T17:29:32.384 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.051076+0000 mon.a (mon.0) 227 : cluster [DBG] min_mon_release 19 (squid) 2024-04-24T17:29:32.384 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.051076+0000 mon.a (mon.0) 227 : cluster [DBG] min_mon_release 19 (squid) 2024-04-24T17:29:32.384 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.051111+0000 mon.a (mon.0) 228 : cluster [DBG] election_strategy: 1 2024-04-24T17:29:32.384 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.051111+0000 mon.a (mon.0) 228 : cluster [DBG] election_strategy: 1 2024-04-24T17:29:32.384 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.051152+0000 mon.a (mon.0) 229 : cluster [DBG] 0: v1:172.21.15.32:6789/0 mon.a 2024-04-24T17:29:32.384 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.051152+0000 mon.a (mon.0) 229 : cluster [DBG] 0: v1:172.21.15.32:6789/0 mon.a 2024-04-24T17:29:32.384 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.051188+0000 mon.a (mon.0) 230 : cluster [DBG] 1: v1:172.21.15.82:6789/0 mon.b 2024-04-24T17:29:32.384 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.051188+0000 mon.a (mon.0) 230 : cluster [DBG] 1: v1:172.21.15.82:6789/0 mon.b 2024-04-24T17:29:32.384 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.051228+0000 mon.a (mon.0) 231 : cluster [DBG] 2: v1:172.21.15.32:6790/0 mon.c 2024-04-24T17:29:32.385 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.051228+0000 mon.a (mon.0) 231 : cluster [DBG] 2: v1:172.21.15.32:6790/0 mon.c 2024-04-24T17:29:32.385 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.071816+0000 mon.a (mon.0) 232 : cluster [DBG] fsmap 2024-04-24T17:29:32.385 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.071816+0000 mon.a (mon.0) 232 : cluster [DBG] fsmap 2024-04-24T17:29:32.385 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.071875+0000 mon.a (mon.0) 233 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-04-24T17:29:32.385 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.071875+0000 mon.a (mon.0) 233 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-04-24T17:29:32.385 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072175+0000 mon.a (mon.0) 234 : cluster [DBG] mgrmap e13: y(active, since 78s) 2024-04-24T17:29:32.385 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072175+0000 mon.a (mon.0) 234 : cluster [DBG] mgrmap e13: y(active, since 78s) 2024-04-24T17:29:32.385 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072457+0000 mon.a (mon.0) 235 : cluster [WRN] Health detail: HEALTH_WARN failed to probe daemons or devices 2024-04-24T17:29:32.385 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072457+0000 mon.a (mon.0) 235 : cluster [WRN] Health detail: HEALTH_WARN failed to probe daemons or devices 2024-04-24T17:29:32.385 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072499+0000 mon.a (mon.0) 236 : cluster [WRN] [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-24T17:29:32.385 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072499+0000 mon.a (mon.0) 236 : cluster [WRN] [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-24T17:29:32.385 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072523+0000 mon.a (mon.0) 237 : cluster [WRN] host smithi082 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T17:29:32.385 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072523+0000 mon.a (mon.0) 237 : cluster [WRN] host smithi082 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T17:29:32.385 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072542+0000 mon.a (mon.0) 238 : cluster [WRN] Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi082 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpbyjftb4d:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch 2024-04-24T17:29:32.385 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072542+0000 mon.a (mon.0) 238 : cluster [WRN] Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi082 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpbyjftb4d:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch 2024-04-24T17:29:32.385 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072562+0000 mon.a (mon.0) 239 : cluster [WRN] /usr/bin/docker: stderr docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:32.386 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072562+0000 mon.a (mon.0) 239 : cluster [WRN] /usr/bin/docker: stderr docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:32.386 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072580+0000 mon.a (mon.0) 240 : cluster [WRN] /usr/bin/docker: stderr See 'docker run --help'. 2024-04-24T17:29:32.386 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072580+0000 mon.a (mon.0) 240 : cluster [WRN] /usr/bin/docker: stderr See 'docker run --help'. 2024-04-24T17:29:32.386 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072599+0000 mon.a (mon.0) 241 : cluster [WRN] Traceback (most recent call last): 2024-04-24T17:29:32.386 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072599+0000 mon.a (mon.0) 241 : cluster [WRN] Traceback (most recent call last): 2024-04-24T17:29:32.386 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072617+0000 mon.a (mon.0) 242 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main 2024-04-24T17:29:32.386 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072617+0000 mon.a (mon.0) 242 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main 2024-04-24T17:29:32.386 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072640+0000 mon.a (mon.0) 243 : cluster [WRN] return _run_code(code, main_globals, None, 2024-04-24T17:29:32.386 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072640+0000 mon.a (mon.0) 243 : cluster [WRN] return _run_code(code, main_globals, None, 2024-04-24T17:29:32.386 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072658+0000 mon.a (mon.0) 244 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 86, in _run_code 2024-04-24T17:29:32.386 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072658+0000 mon.a (mon.0) 244 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 86, in _run_code 2024-04-24T17:29:32.386 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072678+0000 mon.a (mon.0) 245 : cluster [WRN] exec(code, run_globals) 2024-04-24T17:29:32.386 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072678+0000 mon.a (mon.0) 245 : cluster [WRN] exec(code, run_globals) 2024-04-24T17:29:32.386 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072708+0000 mon.a (mon.0) 246 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T17:29:32.387 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072708+0000 mon.a (mon.0) 246 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T17:29:32.387 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072733+0000 mon.a (mon.0) 247 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T17:29:32.387 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072733+0000 mon.a (mon.0) 247 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T17:29:32.387 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072756+0000 mon.a (mon.0) 248 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 408, in _infer_config 2024-04-24T17:29:32.387 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072756+0000 mon.a (mon.0) 248 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 408, in _infer_config 2024-04-24T17:29:32.387 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072778+0000 mon.a (mon.0) 249 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 323, in _infer_fsid 2024-04-24T17:29:32.387 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072778+0000 mon.a (mon.0) 249 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 323, in _infer_fsid 2024-04-24T17:29:32.387 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072805+0000 mon.a (mon.0) 250 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 436, in _infer_image 2024-04-24T17:29:32.387 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072805+0000 mon.a (mon.0) 250 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 436, in _infer_image 2024-04-24T17:29:32.387 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072825+0000 mon.a (mon.0) 251 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 310, in _validate_fsid 2024-04-24T17:29:32.387 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072825+0000 mon.a (mon.0) 251 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 310, in _validate_fsid 2024-04-24T17:29:32.387 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072848+0000 mon.a (mon.0) 252 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3380, in command_ceph_volume 2024-04-24T17:29:32.387 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072848+0000 mon.a (mon.0) 252 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3380, in command_ceph_volume 2024-04-24T17:29:32.387 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072867+0000 mon.a (mon.0) 253 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T17:29:32.387 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072867+0000 mon.a (mon.0) 253 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T17:29:32.388 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072888+0000 mon.a (mon.0) 254 : cluster [WRN] RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi082 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpbyjftb4d:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch: docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:32.388 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072888+0000 mon.a (mon.0) 254 : cluster [WRN] RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi082 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpbyjftb4d:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch: docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:32.388 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072908+0000 mon.a (mon.0) 255 : cluster [WRN] See 'docker run --help'. 2024-04-24T17:29:32.388 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072908+0000 mon.a (mon.0) 255 : cluster [WRN] See 'docker run --help'. 2024-04-24T17:29:32.388 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072926+0000 mon.a (mon.0) 256 : cluster [WRN] host smithi032 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T17:29:32.388 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072926+0000 mon.a (mon.0) 256 : cluster [WRN] host smithi032 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T17:29:32.388 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072947+0000 mon.a (mon.0) 257 : cluster [WRN] Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi032 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpo3z7jlle:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch 2024-04-24T17:29:32.389 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072947+0000 mon.a (mon.0) 257 : cluster [WRN] Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi032 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpo3z7jlle:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch 2024-04-24T17:29:32.389 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072990+0000 mon.a (mon.0) 258 : cluster [WRN] /usr/bin/docker: stderr docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:32.389 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.072990+0000 mon.a (mon.0) 258 : cluster [WRN] /usr/bin/docker: stderr docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:32.389 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.073009+0000 mon.a (mon.0) 259 : cluster [WRN] /usr/bin/docker: stderr See 'docker run --help'. 2024-04-24T17:29:32.389 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.073009+0000 mon.a (mon.0) 259 : cluster [WRN] /usr/bin/docker: stderr See 'docker run --help'. 2024-04-24T17:29:32.389 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.073027+0000 mon.a (mon.0) 260 : cluster [WRN] Traceback (most recent call last): 2024-04-24T17:29:32.389 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.073027+0000 mon.a (mon.0) 260 : cluster [WRN] Traceback (most recent call last): 2024-04-24T17:29:32.389 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.073047+0000 mon.a (mon.0) 261 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main 2024-04-24T17:29:32.389 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.073047+0000 mon.a (mon.0) 261 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main 2024-04-24T17:29:32.389 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.073067+0000 mon.a (mon.0) 262 : cluster [WRN] return _run_code(code, main_globals, None, 2024-04-24T17:29:32.389 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.073067+0000 mon.a (mon.0) 262 : cluster [WRN] return _run_code(code, main_globals, None, 2024-04-24T17:29:32.389 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.073086+0000 mon.a (mon.0) 263 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 86, in _run_code 2024-04-24T17:29:32.389 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.073086+0000 mon.a (mon.0) 263 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 86, in _run_code 2024-04-24T17:29:32.389 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.073104+0000 mon.a (mon.0) 264 : cluster [WRN] exec(code, run_globals) 2024-04-24T17:29:32.390 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.073104+0000 mon.a (mon.0) 264 : cluster [WRN] exec(code, run_globals) 2024-04-24T17:29:32.390 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.073126+0000 mon.a (mon.0) 265 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T17:29:32.390 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.073126+0000 mon.a (mon.0) 265 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T17:29:32.390 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.073144+0000 mon.a (mon.0) 266 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T17:29:32.390 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.073144+0000 mon.a (mon.0) 266 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T17:29:32.390 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.073165+0000 mon.a (mon.0) 267 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 408, in _infer_config 2024-04-24T17:29:32.390 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.073165+0000 mon.a (mon.0) 267 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 408, in _infer_config 2024-04-24T17:29:32.390 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.073183+0000 mon.a (mon.0) 268 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 323, in _infer_fsid 2024-04-24T17:29:32.390 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.073183+0000 mon.a (mon.0) 268 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 323, in _infer_fsid 2024-04-24T17:29:32.390 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.073201+0000 mon.a (mon.0) 269 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 436, in _infer_image 2024-04-24T17:29:32.390 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.073201+0000 mon.a (mon.0) 269 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 436, in _infer_image 2024-04-24T17:29:32.390 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.073221+0000 mon.a (mon.0) 270 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 310, in _validate_fsid 2024-04-24T17:29:32.390 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.073221+0000 mon.a (mon.0) 270 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 310, in _validate_fsid 2024-04-24T17:29:32.390 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.073241+0000 mon.a (mon.0) 271 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3380, in command_ceph_volume 2024-04-24T17:29:32.391 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.073241+0000 mon.a (mon.0) 271 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3380, in command_ceph_volume 2024-04-24T17:29:32.391 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.073259+0000 mon.a (mon.0) 272 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T17:29:32.391 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.073259+0000 mon.a (mon.0) 272 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T17:29:32.391 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.073282+0000 mon.a (mon.0) 273 : cluster [WRN] RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi032 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpo3z7jlle:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch: docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:32.391 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.073282+0000 mon.a (mon.0) 273 : cluster [WRN] RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi032 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpo3z7jlle:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch: docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:32.391 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.073301+0000 mon.a (mon.0) 274 : cluster [WRN] See 'docker run --help'. 2024-04-24T17:29:32.391 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: cluster 2024-04-24T17:29:32.073301+0000 mon.a (mon.0) 274 : cluster [WRN] See 'docker run --help'. 2024-04-24T17:29:32.391 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: audit 2024-04-24T17:29:32.083395+0000 mon.a (mon.0) 275 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:32.391 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: audit 2024-04-24T17:29:32.083395+0000 mon.a (mon.0) 275 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:32.391 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: audit 2024-04-24T17:29:32.089321+0000 mon.a (mon.0) 276 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:32.391 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: audit 2024-04-24T17:29:32.089321+0000 mon.a (mon.0) 276 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:32.391 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: audit 2024-04-24T17:29:32.096440+0000 mon.a (mon.0) 277 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:32.391 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:32 smithi082 bash[21793]: audit 2024-04-24T17:29:32.096440+0000 mon.a (mon.0) 277 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:32.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: audit 2024-04-24T17:29:26.998218+0000 mon.a (mon.0) 212 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T17:29:32.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: audit 2024-04-24T17:29:26.998218+0000 mon.a (mon.0) 212 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T17:29:32.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.998654+0000 mon.a (mon.0) 213 : cluster [INF] mon.a calling monitor election 2024-04-24T17:29:32.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:26.998654+0000 mon.a (mon.0) 213 : cluster [INF] mon.a calling monitor election 2024-04-24T17:29:32.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:27.003312+0000 mon.b (mon.1) 2 : cluster [INF] mon.b calling monitor election 2024-04-24T17:29:32.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:27.003312+0000 mon.b (mon.1) 2 : cluster [INF] mon.b calling monitor election 2024-04-24T17:29:32.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: audit 2024-04-24T17:29:27.020403+0000 mon.a (mon.0) 214 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T17:29:32.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: audit 2024-04-24T17:29:27.020403+0000 mon.a (mon.0) 214 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T17:29:32.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: audit 2024-04-24T17:29:27.020738+0000 mon.a (mon.0) 215 : audit [DBG] from='client.? v1:172.21.15.82:0/2538690701' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T17:29:32.504 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: audit 2024-04-24T17:29:27.020738+0000 mon.a (mon.0) 215 : audit [DBG] from='client.? v1:172.21.15.82:0/2538690701' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T17:29:32.504 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: audit 2024-04-24T17:29:27.020961+0000 mon.a (mon.0) 216 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:32.504 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: audit 2024-04-24T17:29:27.020961+0000 mon.a (mon.0) 216 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:32.504 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:27.040410+0000 mgr.y (mgr.14152) 60 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:32.504 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:27.040410+0000 mgr.y (mgr.14152) 60 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:32.504 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: audit 2024-04-24T17:29:27.990231+0000 mon.a (mon.0) 217 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:32.504 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: audit 2024-04-24T17:29:27.990231+0000 mon.a (mon.0) 217 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:32.504 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: audit 2024-04-24T17:29:28.990307+0000 mon.a (mon.0) 218 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:32.504 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: audit 2024-04-24T17:29:28.990307+0000 mon.a (mon.0) 218 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:32.504 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:28.993624+0000 mon.c (mon.2) 1 : cluster [INF] mon.c calling monitor election 2024-04-24T17:29:32.504 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:28.993624+0000 mon.c (mon.2) 1 : cluster [INF] mon.c calling monitor election 2024-04-24T17:29:32.504 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:29.041381+0000 mgr.y (mgr.14152) 61 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:32.505 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:21.037808+0000 mgr.y (mgr.14152) 57 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:32.505 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:21.037808+0000 mgr.y (mgr.14152) 57 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:32.505 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:21.178426+0000 mon.a (mon.0) 145 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T17:29:32.505 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:21.178426+0000 mon.a (mon.0) 145 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T17:29:32.505 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:21.178586+0000 mon.a (mon.0) 146 : cluster [INF] mon.a calling monitor election 2024-04-24T17:29:32.505 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:21.178586+0000 mon.a (mon.0) 146 : cluster [INF] mon.a calling monitor election 2024-04-24T17:29:32.505 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:21.199736+0000 mon.a (mon.0) 147 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T17:29:32.505 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:21.199736+0000 mon.a (mon.0) 147 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T17:29:32.505 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:22.174508+0000 mon.a (mon.0) 148 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T17:29:32.505 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:22.174508+0000 mon.a (mon.0) 148 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T17:29:32.505 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:23.038793+0000 mgr.y (mgr.14152) 58 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:32.506 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:23.038793+0000 mgr.y (mgr.14152) 58 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:32.506 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:23.174606+0000 mon.a (mon.0) 149 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T17:29:32.506 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:23.174606+0000 mon.a (mon.0) 149 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T17:29:32.506 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:23.178079+0000 mon.b (mon.1) 1 : cluster [INF] mon.b calling monitor election 2024-04-24T17:29:32.506 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:23.178079+0000 mon.b (mon.1) 1 : cluster [INF] mon.b calling monitor election 2024-04-24T17:29:32.506 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:24.174708+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T17:29:32.506 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:24.174708+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T17:29:32.506 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:24.989729+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:32.506 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:24.989729+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:32.506 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:25.039437+0000 mgr.y (mgr.14152) 59 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:32.506 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:25.039437+0000 mgr.y (mgr.14152) 59 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:32.506 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:25.175022+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T17:29:32.506 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:25.175022+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T17:29:32.506 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:25.989853+0000 mon.a (mon.0) 153 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:32.507 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:25.989853+0000 mon.a (mon.0) 153 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:32.507 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:26.175159+0000 mon.a (mon.0) 154 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T17:29:32.507 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:26.175159+0000 mon.a (mon.0) 154 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T17:29:32.507 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.222801+0000 mon.a (mon.0) 155 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-04-24T17:29:32.507 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.222801+0000 mon.a (mon.0) 155 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-04-24T17:29:32.507 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.228764+0000 mon.a (mon.0) 156 : cluster [DBG] monmap epoch 2 2024-04-24T17:29:32.507 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.228764+0000 mon.a (mon.0) 156 : cluster [DBG] monmap epoch 2 2024-04-24T17:29:32.507 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.228825+0000 mon.a (mon.0) 157 : cluster [DBG] fsid a7981b84-025f-11ef-bc93-c7b262605968 2024-04-24T17:29:32.507 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.228825+0000 mon.a (mon.0) 157 : cluster [DBG] fsid a7981b84-025f-11ef-bc93-c7b262605968 2024-04-24T17:29:32.507 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.228843+0000 mon.a (mon.0) 158 : cluster [DBG] last_changed 2024-04-24T17:29:21.175737+0000 2024-04-24T17:29:32.507 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.228843+0000 mon.a (mon.0) 158 : cluster [DBG] last_changed 2024-04-24T17:29:21.175737+0000 2024-04-24T17:29:32.507 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.228865+0000 mon.a (mon.0) 159 : cluster [DBG] created 2024-04-24T17:26:59.605652+0000 2024-04-24T17:29:32.507 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.228865+0000 mon.a (mon.0) 159 : cluster [DBG] created 2024-04-24T17:26:59.605652+0000 2024-04-24T17:29:32.507 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.228888+0000 mon.a (mon.0) 160 : cluster [DBG] min_mon_release 19 (squid) 2024-04-24T17:29:32.507 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.228888+0000 mon.a (mon.0) 160 : cluster [DBG] min_mon_release 19 (squid) 2024-04-24T17:29:32.508 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.228908+0000 mon.a (mon.0) 161 : cluster [DBG] election_strategy: 1 2024-04-24T17:29:32.508 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.228908+0000 mon.a (mon.0) 161 : cluster [DBG] election_strategy: 1 2024-04-24T17:29:32.508 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.228927+0000 mon.a (mon.0) 162 : cluster [DBG] 0: v1:172.21.15.32:6789/0 mon.a 2024-04-24T17:29:32.508 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.228927+0000 mon.a (mon.0) 162 : cluster [DBG] 0: v1:172.21.15.32:6789/0 mon.a 2024-04-24T17:29:32.508 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.228945+0000 mon.a (mon.0) 163 : cluster [DBG] 1: v1:172.21.15.82:6789/0 mon.b 2024-04-24T17:29:32.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:29.041381+0000 mgr.y (mgr.14152) 61 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:32.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: audit 2024-04-24T17:29:29.990698+0000 mon.a (mon.0) 219 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:32.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: audit 2024-04-24T17:29:29.990698+0000 mon.a (mon.0) 219 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:32.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: audit 2024-04-24T17:29:30.990765+0000 mon.a (mon.0) 220 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:32.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: audit 2024-04-24T17:29:30.990765+0000 mon.a (mon.0) 220 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:32.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:31.042310+0000 mgr.y (mgr.14152) 62 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:32.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:31.042310+0000 mgr.y (mgr.14152) 62 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:32.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: audit 2024-04-24T17:29:31.990982+0000 mon.a (mon.0) 221 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:32.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: audit 2024-04-24T17:29:31.990982+0000 mon.a (mon.0) 221 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:32.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.042895+0000 mon.a (mon.0) 222 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-04-24T17:29:32.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.042895+0000 mon.a (mon.0) 222 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-04-24T17:29:32.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.050928+0000 mon.a (mon.0) 223 : cluster [DBG] monmap epoch 3 2024-04-24T17:29:32.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.050928+0000 mon.a (mon.0) 223 : cluster [DBG] monmap epoch 3 2024-04-24T17:29:32.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.050983+0000 mon.a (mon.0) 224 : cluster [DBG] fsid a7981b84-025f-11ef-bc93-c7b262605968 2024-04-24T17:29:32.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.050983+0000 mon.a (mon.0) 224 : cluster [DBG] fsid a7981b84-025f-11ef-bc93-c7b262605968 2024-04-24T17:29:32.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.051029+0000 mon.a (mon.0) 225 : cluster [DBG] last_changed 2024-04-24T17:29:26.991555+0000 2024-04-24T17:29:32.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.051029+0000 mon.a (mon.0) 225 : cluster [DBG] last_changed 2024-04-24T17:29:26.991555+0000 2024-04-24T17:29:32.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.051052+0000 mon.a (mon.0) 226 : cluster [DBG] created 2024-04-24T17:26:59.605652+0000 2024-04-24T17:29:32.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.051052+0000 mon.a (mon.0) 226 : cluster [DBG] created 2024-04-24T17:26:59.605652+0000 2024-04-24T17:29:32.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.051076+0000 mon.a (mon.0) 227 : cluster [DBG] min_mon_release 19 (squid) 2024-04-24T17:29:32.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.051076+0000 mon.a (mon.0) 227 : cluster [DBG] min_mon_release 19 (squid) 2024-04-24T17:29:32.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.051111+0000 mon.a (mon.0) 228 : cluster [DBG] election_strategy: 1 2024-04-24T17:29:32.509 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.051111+0000 mon.a (mon.0) 228 : cluster [DBG] election_strategy: 1 2024-04-24T17:29:32.510 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.051152+0000 mon.a (mon.0) 229 : cluster [DBG] 0: v1:172.21.15.32:6789/0 mon.a 2024-04-24T17:29:32.510 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.228945+0000 mon.a (mon.0) 163 : cluster [DBG] 1: v1:172.21.15.82:6789/0 mon.b 2024-04-24T17:29:32.510 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.247828+0000 mon.a (mon.0) 164 : cluster [DBG] fsmap 2024-04-24T17:29:32.510 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.247828+0000 mon.a (mon.0) 164 : cluster [DBG] fsmap 2024-04-24T17:29:32.510 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.247881+0000 mon.a (mon.0) 165 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-04-24T17:29:32.510 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.247881+0000 mon.a (mon.0) 165 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-04-24T17:29:32.510 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248156+0000 mon.a (mon.0) 166 : cluster [DBG] mgrmap e13: y(active, since 72s) 2024-04-24T17:29:32.510 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248156+0000 mon.a (mon.0) 166 : cluster [DBG] mgrmap e13: y(active, since 72s) 2024-04-24T17:29:32.510 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248446+0000 mon.a (mon.0) 167 : cluster [WRN] Health detail: HEALTH_WARN failed to probe daemons or devices 2024-04-24T17:29:32.510 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248446+0000 mon.a (mon.0) 167 : cluster [WRN] Health detail: HEALTH_WARN failed to probe daemons or devices 2024-04-24T17:29:32.511 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248480+0000 mon.a (mon.0) 168 : cluster [WRN] [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-24T17:29:32.511 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248480+0000 mon.a (mon.0) 168 : cluster [WRN] [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-24T17:29:32.511 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248496+0000 mon.a (mon.0) 169 : cluster [WRN] host smithi082 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T17:29:32.511 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248496+0000 mon.a (mon.0) 169 : cluster [WRN] host smithi082 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T17:29:32.511 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248513+0000 mon.a (mon.0) 170 : cluster [WRN] Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi082 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpbyjftb4d:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch 2024-04-24T17:29:32.511 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248513+0000 mon.a (mon.0) 170 : cluster [WRN] Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi082 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpbyjftb4d:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch 2024-04-24T17:29:32.511 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.051152+0000 mon.a (mon.0) 229 : cluster [DBG] 0: v1:172.21.15.32:6789/0 mon.a 2024-04-24T17:29:32.511 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.051188+0000 mon.a (mon.0) 230 : cluster [DBG] 1: v1:172.21.15.82:6789/0 mon.b 2024-04-24T17:29:32.511 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.051188+0000 mon.a (mon.0) 230 : cluster [DBG] 1: v1:172.21.15.82:6789/0 mon.b 2024-04-24T17:29:32.511 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.051228+0000 mon.a (mon.0) 231 : cluster [DBG] 2: v1:172.21.15.32:6790/0 mon.c 2024-04-24T17:29:32.511 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.051228+0000 mon.a (mon.0) 231 : cluster [DBG] 2: v1:172.21.15.32:6790/0 mon.c 2024-04-24T17:29:32.511 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.071816+0000 mon.a (mon.0) 232 : cluster [DBG] fsmap 2024-04-24T17:29:32.512 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.071816+0000 mon.a (mon.0) 232 : cluster [DBG] fsmap 2024-04-24T17:29:32.512 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.071875+0000 mon.a (mon.0) 233 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-04-24T17:29:32.512 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.071875+0000 mon.a (mon.0) 233 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-04-24T17:29:32.512 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072175+0000 mon.a (mon.0) 234 : cluster [DBG] mgrmap e13: y(active, since 78s) 2024-04-24T17:29:32.512 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072175+0000 mon.a (mon.0) 234 : cluster [DBG] mgrmap e13: y(active, since 78s) 2024-04-24T17:29:32.512 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072457+0000 mon.a (mon.0) 235 : cluster [WRN] Health detail: HEALTH_WARN failed to probe daemons or devices 2024-04-24T17:29:32.512 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072457+0000 mon.a (mon.0) 235 : cluster [WRN] Health detail: HEALTH_WARN failed to probe daemons or devices 2024-04-24T17:29:32.512 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072499+0000 mon.a (mon.0) 236 : cluster [WRN] [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-24T17:29:32.512 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072499+0000 mon.a (mon.0) 236 : cluster [WRN] [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-24T17:29:32.512 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072523+0000 mon.a (mon.0) 237 : cluster [WRN] host smithi082 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T17:29:32.512 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072523+0000 mon.a (mon.0) 237 : cluster [WRN] host smithi082 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T17:29:32.512 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072542+0000 mon.a (mon.0) 238 : cluster [WRN] Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi082 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpbyjftb4d:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch 2024-04-24T17:29:32.513 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248528+0000 mon.a (mon.0) 171 : cluster [WRN] /usr/bin/docker: stderr docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:32.513 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248528+0000 mon.a (mon.0) 171 : cluster [WRN] /usr/bin/docker: stderr docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:32.513 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248547+0000 mon.a (mon.0) 172 : cluster [WRN] /usr/bin/docker: stderr See 'docker run --help'. 2024-04-24T17:29:32.513 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248547+0000 mon.a (mon.0) 172 : cluster [WRN] /usr/bin/docker: stderr See 'docker run --help'. 2024-04-24T17:29:32.513 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248565+0000 mon.a (mon.0) 173 : cluster [WRN] Traceback (most recent call last): 2024-04-24T17:29:32.513 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248565+0000 mon.a (mon.0) 173 : cluster [WRN] Traceback (most recent call last): 2024-04-24T17:29:32.513 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248587+0000 mon.a (mon.0) 174 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main 2024-04-24T17:29:32.513 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248587+0000 mon.a (mon.0) 174 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main 2024-04-24T17:29:32.513 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248606+0000 mon.a (mon.0) 175 : cluster [WRN] return _run_code(code, main_globals, None, 2024-04-24T17:29:32.513 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248606+0000 mon.a (mon.0) 175 : cluster [WRN] return _run_code(code, main_globals, None, 2024-04-24T17:29:32.513 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248623+0000 mon.a (mon.0) 176 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 86, in _run_code 2024-04-24T17:29:32.513 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248623+0000 mon.a (mon.0) 176 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 86, in _run_code 2024-04-24T17:29:32.513 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248643+0000 mon.a (mon.0) 177 : cluster [WRN] exec(code, run_globals) 2024-04-24T17:29:32.514 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248643+0000 mon.a (mon.0) 177 : cluster [WRN] exec(code, run_globals) 2024-04-24T17:29:32.514 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248661+0000 mon.a (mon.0) 178 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T17:29:32.514 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248661+0000 mon.a (mon.0) 178 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T17:29:32.514 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248681+0000 mon.a (mon.0) 179 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T17:29:32.514 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248681+0000 mon.a (mon.0) 179 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T17:29:32.514 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248699+0000 mon.a (mon.0) 180 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 408, in _infer_config 2024-04-24T17:29:32.514 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248699+0000 mon.a (mon.0) 180 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 408, in _infer_config 2024-04-24T17:29:32.514 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248718+0000 mon.a (mon.0) 181 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 323, in _infer_fsid 2024-04-24T17:29:32.514 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248718+0000 mon.a (mon.0) 181 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 323, in _infer_fsid 2024-04-24T17:29:32.514 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248738+0000 mon.a (mon.0) 182 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 436, in _infer_image 2024-04-24T17:29:32.514 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248738+0000 mon.a (mon.0) 182 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 436, in _infer_image 2024-04-24T17:29:32.514 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248757+0000 mon.a (mon.0) 183 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 310, in _validate_fsid 2024-04-24T17:29:32.514 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248757+0000 mon.a (mon.0) 183 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 310, in _validate_fsid 2024-04-24T17:29:32.514 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248780+0000 mon.a (mon.0) 184 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3380, in command_ceph_volume 2024-04-24T17:29:32.515 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248780+0000 mon.a (mon.0) 184 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3380, in command_ceph_volume 2024-04-24T17:29:32.515 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248803+0000 mon.a (mon.0) 185 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T17:29:32.515 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248803+0000 mon.a (mon.0) 185 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T17:29:32.515 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072542+0000 mon.a (mon.0) 238 : cluster [WRN] Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi082 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpbyjftb4d:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch 2024-04-24T17:29:32.515 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072562+0000 mon.a (mon.0) 239 : cluster [WRN] /usr/bin/docker: stderr docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:32.515 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072562+0000 mon.a (mon.0) 239 : cluster [WRN] /usr/bin/docker: stderr docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:32.515 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072580+0000 mon.a (mon.0) 240 : cluster [WRN] /usr/bin/docker: stderr See 'docker run --help'. 2024-04-24T17:29:32.515 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072580+0000 mon.a (mon.0) 240 : cluster [WRN] /usr/bin/docker: stderr See 'docker run --help'. 2024-04-24T17:29:32.515 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072599+0000 mon.a (mon.0) 241 : cluster [WRN] Traceback (most recent call last): 2024-04-24T17:29:32.515 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072599+0000 mon.a (mon.0) 241 : cluster [WRN] Traceback (most recent call last): 2024-04-24T17:29:32.515 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072617+0000 mon.a (mon.0) 242 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main 2024-04-24T17:29:32.516 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072617+0000 mon.a (mon.0) 242 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main 2024-04-24T17:29:32.516 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072640+0000 mon.a (mon.0) 243 : cluster [WRN] return _run_code(code, main_globals, None, 2024-04-24T17:29:32.516 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072640+0000 mon.a (mon.0) 243 : cluster [WRN] return _run_code(code, main_globals, None, 2024-04-24T17:29:32.516 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072658+0000 mon.a (mon.0) 244 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 86, in _run_code 2024-04-24T17:29:32.516 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072658+0000 mon.a (mon.0) 244 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 86, in _run_code 2024-04-24T17:29:32.516 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072678+0000 mon.a (mon.0) 245 : cluster [WRN] exec(code, run_globals) 2024-04-24T17:29:32.516 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072678+0000 mon.a (mon.0) 245 : cluster [WRN] exec(code, run_globals) 2024-04-24T17:29:32.516 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072708+0000 mon.a (mon.0) 246 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T17:29:32.516 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072708+0000 mon.a (mon.0) 246 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T17:29:32.516 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248821+0000 mon.a (mon.0) 186 : cluster [WRN] RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi082 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpbyjftb4d:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch: docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:32.516 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248821+0000 mon.a (mon.0) 186 : cluster [WRN] RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi082 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpbyjftb4d:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch: docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:32.517 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248846+0000 mon.a (mon.0) 187 : cluster [WRN] See 'docker run --help'. 2024-04-24T17:29:32.517 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248846+0000 mon.a (mon.0) 187 : cluster [WRN] See 'docker run --help'. 2024-04-24T17:29:32.517 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248865+0000 mon.a (mon.0) 188 : cluster [WRN] host smithi032 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T17:29:32.517 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248865+0000 mon.a (mon.0) 188 : cluster [WRN] host smithi032 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T17:29:32.517 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248883+0000 mon.a (mon.0) 189 : cluster [WRN] Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi032 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpo3z7jlle:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch 2024-04-24T17:29:32.517 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072733+0000 mon.a (mon.0) 247 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T17:29:32.517 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072733+0000 mon.a (mon.0) 247 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T17:29:32.517 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072756+0000 mon.a (mon.0) 248 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 408, in _infer_config 2024-04-24T17:29:32.517 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072756+0000 mon.a (mon.0) 248 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 408, in _infer_config 2024-04-24T17:29:32.517 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072778+0000 mon.a (mon.0) 249 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 323, in _infer_fsid 2024-04-24T17:29:32.517 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072778+0000 mon.a (mon.0) 249 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 323, in _infer_fsid 2024-04-24T17:29:32.517 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072805+0000 mon.a (mon.0) 250 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 436, in _infer_image 2024-04-24T17:29:32.517 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072805+0000 mon.a (mon.0) 250 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 436, in _infer_image 2024-04-24T17:29:32.518 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072825+0000 mon.a (mon.0) 251 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 310, in _validate_fsid 2024-04-24T17:29:32.518 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072825+0000 mon.a (mon.0) 251 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 310, in _validate_fsid 2024-04-24T17:29:32.518 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072848+0000 mon.a (mon.0) 252 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3380, in command_ceph_volume 2024-04-24T17:29:32.518 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072848+0000 mon.a (mon.0) 252 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3380, in command_ceph_volume 2024-04-24T17:29:32.518 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072867+0000 mon.a (mon.0) 253 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T17:29:32.518 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072867+0000 mon.a (mon.0) 253 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T17:29:32.518 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248883+0000 mon.a (mon.0) 189 : cluster [WRN] Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi032 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpo3z7jlle:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch 2024-04-24T17:29:32.518 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248908+0000 mon.a (mon.0) 190 : cluster [WRN] /usr/bin/docker: stderr docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:32.518 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248908+0000 mon.a (mon.0) 190 : cluster [WRN] /usr/bin/docker: stderr docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:32.519 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248926+0000 mon.a (mon.0) 191 : cluster [WRN] /usr/bin/docker: stderr See 'docker run --help'. 2024-04-24T17:29:32.519 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248926+0000 mon.a (mon.0) 191 : cluster [WRN] /usr/bin/docker: stderr See 'docker run --help'. 2024-04-24T17:29:32.519 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248943+0000 mon.a (mon.0) 192 : cluster [WRN] Traceback (most recent call last): 2024-04-24T17:29:32.519 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248943+0000 mon.a (mon.0) 192 : cluster [WRN] Traceback (most recent call last): 2024-04-24T17:29:32.519 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248961+0000 mon.a (mon.0) 193 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main 2024-04-24T17:29:32.519 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248961+0000 mon.a (mon.0) 193 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main 2024-04-24T17:29:32.519 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248982+0000 mon.a (mon.0) 194 : cluster [WRN] return _run_code(code, main_globals, None, 2024-04-24T17:29:32.519 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.248982+0000 mon.a (mon.0) 194 : cluster [WRN] return _run_code(code, main_globals, None, 2024-04-24T17:29:32.519 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.249000+0000 mon.a (mon.0) 195 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 86, in _run_code 2024-04-24T17:29:32.519 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.249000+0000 mon.a (mon.0) 195 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 86, in _run_code 2024-04-24T17:29:32.519 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.249019+0000 mon.a (mon.0) 196 : cluster [WRN] exec(code, run_globals) 2024-04-24T17:29:32.519 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.249019+0000 mon.a (mon.0) 196 : cluster [WRN] exec(code, run_globals) 2024-04-24T17:29:32.519 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.249037+0000 mon.a (mon.0) 197 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T17:29:32.519 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.249037+0000 mon.a (mon.0) 197 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T17:29:32.519 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.249055+0000 mon.a (mon.0) 198 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T17:29:32.520 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072888+0000 mon.a (mon.0) 254 : cluster [WRN] RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi082 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpbyjftb4d:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch: docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:32.520 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072888+0000 mon.a (mon.0) 254 : cluster [WRN] RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi082 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpbyjftb4d:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch: docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:32.520 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072908+0000 mon.a (mon.0) 255 : cluster [WRN] See 'docker run --help'. 2024-04-24T17:29:32.520 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072908+0000 mon.a (mon.0) 255 : cluster [WRN] See 'docker run --help'. 2024-04-24T17:29:32.520 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072926+0000 mon.a (mon.0) 256 : cluster [WRN] host smithi032 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T17:29:32.520 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072926+0000 mon.a (mon.0) 256 : cluster [WRN] host smithi032 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T17:29:32.520 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072947+0000 mon.a (mon.0) 257 : cluster [WRN] Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi032 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpo3z7jlle:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch 2024-04-24T17:29:32.521 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.249055+0000 mon.a (mon.0) 198 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T17:29:32.521 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.249074+0000 mon.a (mon.0) 199 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 408, in _infer_config 2024-04-24T17:29:32.521 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.249074+0000 mon.a (mon.0) 199 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 408, in _infer_config 2024-04-24T17:29:32.521 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.249099+0000 mon.a (mon.0) 200 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 323, in _infer_fsid 2024-04-24T17:29:32.521 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.249099+0000 mon.a (mon.0) 200 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 323, in _infer_fsid 2024-04-24T17:29:32.521 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.249118+0000 mon.a (mon.0) 201 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 436, in _infer_image 2024-04-24T17:29:32.521 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.249118+0000 mon.a (mon.0) 201 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 436, in _infer_image 2024-04-24T17:29:32.521 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.249139+0000 mon.a (mon.0) 202 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 310, in _validate_fsid 2024-04-24T17:29:32.521 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.249139+0000 mon.a (mon.0) 202 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 310, in _validate_fsid 2024-04-24T17:29:32.521 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.249158+0000 mon.a (mon.0) 203 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3380, in command_ceph_volume 2024-04-24T17:29:32.521 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.249158+0000 mon.a (mon.0) 203 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3380, in command_ceph_volume 2024-04-24T17:29:32.521 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.249178+0000 mon.a (mon.0) 204 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T17:29:32.522 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.249178+0000 mon.a (mon.0) 204 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T17:29:32.522 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.249199+0000 mon.a (mon.0) 205 : cluster [WRN] RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi032 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpo3z7jlle:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch: docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:32.522 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.249199+0000 mon.a (mon.0) 205 : cluster [WRN] RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi032 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpo3z7jlle:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch: docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:32.522 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.249227+0000 mon.a (mon.0) 206 : cluster [WRN] See 'docker run --help'. 2024-04-24T17:29:32.522 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.249227+0000 mon.a (mon.0) 206 : cluster [WRN] See 'docker run --help'. 2024-04-24T17:29:32.522 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:26.256924+0000 mon.a (mon.0) 207 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:32.522 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:26.256924+0000 mon.a (mon.0) 207 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:32.522 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:26.272804+0000 mon.a (mon.0) 208 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:32.522 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:26.272804+0000 mon.a (mon.0) 208 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:32.522 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:26.280157+0000 mon.a (mon.0) 209 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:32.522 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:26.280157+0000 mon.a (mon.0) 209 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:32.522 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:26.998218+0000 mon.a (mon.0) 212 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T17:29:32.522 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:26.998218+0000 mon.a (mon.0) 212 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T17:29:32.522 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.998654+0000 mon.a (mon.0) 213 : cluster [INF] mon.a calling monitor election 2024-04-24T17:29:32.523 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072947+0000 mon.a (mon.0) 257 : cluster [WRN] Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi032 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpo3z7jlle:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch 2024-04-24T17:29:32.523 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072990+0000 mon.a (mon.0) 258 : cluster [WRN] /usr/bin/docker: stderr docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:32.523 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.072990+0000 mon.a (mon.0) 258 : cluster [WRN] /usr/bin/docker: stderr docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:32.523 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.073009+0000 mon.a (mon.0) 259 : cluster [WRN] /usr/bin/docker: stderr See 'docker run --help'. 2024-04-24T17:29:32.523 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.073009+0000 mon.a (mon.0) 259 : cluster [WRN] /usr/bin/docker: stderr See 'docker run --help'. 2024-04-24T17:29:32.523 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.073027+0000 mon.a (mon.0) 260 : cluster [WRN] Traceback (most recent call last): 2024-04-24T17:29:32.523 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.073027+0000 mon.a (mon.0) 260 : cluster [WRN] Traceback (most recent call last): 2024-04-24T17:29:32.523 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.073047+0000 mon.a (mon.0) 261 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main 2024-04-24T17:29:32.523 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.073047+0000 mon.a (mon.0) 261 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main 2024-04-24T17:29:32.523 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.073067+0000 mon.a (mon.0) 262 : cluster [WRN] return _run_code(code, main_globals, None, 2024-04-24T17:29:32.523 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.073067+0000 mon.a (mon.0) 262 : cluster [WRN] return _run_code(code, main_globals, None, 2024-04-24T17:29:32.523 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.073086+0000 mon.a (mon.0) 263 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 86, in _run_code 2024-04-24T17:29:32.524 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.073086+0000 mon.a (mon.0) 263 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 86, in _run_code 2024-04-24T17:29:32.524 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.073104+0000 mon.a (mon.0) 264 : cluster [WRN] exec(code, run_globals) 2024-04-24T17:29:32.524 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.073104+0000 mon.a (mon.0) 264 : cluster [WRN] exec(code, run_globals) 2024-04-24T17:29:32.524 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.073126+0000 mon.a (mon.0) 265 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T17:29:32.524 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:26.998654+0000 mon.a (mon.0) 213 : cluster [INF] mon.a calling monitor election 2024-04-24T17:29:32.524 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:27.003312+0000 mon.b (mon.1) 2 : cluster [INF] mon.b calling monitor election 2024-04-24T17:29:32.524 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:27.003312+0000 mon.b (mon.1) 2 : cluster [INF] mon.b calling monitor election 2024-04-24T17:29:32.524 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:27.020403+0000 mon.a (mon.0) 214 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T17:29:32.524 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:27.020403+0000 mon.a (mon.0) 214 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T17:29:32.524 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:27.020738+0000 mon.a (mon.0) 215 : audit [DBG] from='client.? v1:172.21.15.82:0/2538690701' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T17:29:32.524 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:27.020738+0000 mon.a (mon.0) 215 : audit [DBG] from='client.? v1:172.21.15.82:0/2538690701' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T17:29:32.524 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:27.020961+0000 mon.a (mon.0) 216 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:32.525 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:27.020961+0000 mon.a (mon.0) 216 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:32.525 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:27.040410+0000 mgr.y (mgr.14152) 60 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:32.525 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:27.040410+0000 mgr.y (mgr.14152) 60 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:32.525 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:27.990231+0000 mon.a (mon.0) 217 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:32.525 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:27.990231+0000 mon.a (mon.0) 217 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:32.525 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:28.990307+0000 mon.a (mon.0) 218 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:32.525 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:28.990307+0000 mon.a (mon.0) 218 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:32.525 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:28.993624+0000 mon.c (mon.2) 1 : cluster [INF] mon.c calling monitor election 2024-04-24T17:29:32.525 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:28.993624+0000 mon.c (mon.2) 1 : cluster [INF] mon.c calling monitor election 2024-04-24T17:29:32.525 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:29.041381+0000 mgr.y (mgr.14152) 61 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:32.525 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:29.041381+0000 mgr.y (mgr.14152) 61 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:32.525 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:29.990698+0000 mon.a (mon.0) 219 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:32.525 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:29.990698+0000 mon.a (mon.0) 219 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:32.526 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.073126+0000 mon.a (mon.0) 265 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T17:29:32.526 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.073144+0000 mon.a (mon.0) 266 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T17:29:32.526 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.073144+0000 mon.a (mon.0) 266 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T17:29:32.526 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.073165+0000 mon.a (mon.0) 267 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 408, in _infer_config 2024-04-24T17:29:32.526 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.073165+0000 mon.a (mon.0) 267 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 408, in _infer_config 2024-04-24T17:29:32.526 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.073183+0000 mon.a (mon.0) 268 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 323, in _infer_fsid 2024-04-24T17:29:32.526 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.073183+0000 mon.a (mon.0) 268 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 323, in _infer_fsid 2024-04-24T17:29:32.526 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.073201+0000 mon.a (mon.0) 269 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 436, in _infer_image 2024-04-24T17:29:32.526 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.073201+0000 mon.a (mon.0) 269 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 436, in _infer_image 2024-04-24T17:29:32.527 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.073221+0000 mon.a (mon.0) 270 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 310, in _validate_fsid 2024-04-24T17:29:32.527 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.073221+0000 mon.a (mon.0) 270 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 310, in _validate_fsid 2024-04-24T17:29:32.527 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.073241+0000 mon.a (mon.0) 271 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3380, in command_ceph_volume 2024-04-24T17:29:32.527 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.073241+0000 mon.a (mon.0) 271 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3380, in command_ceph_volume 2024-04-24T17:29:32.527 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.073259+0000 mon.a (mon.0) 272 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T17:29:32.527 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.073259+0000 mon.a (mon.0) 272 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T17:29:32.527 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.073282+0000 mon.a (mon.0) 273 : cluster [WRN] RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi032 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpo3z7jlle:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch: docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:32.527 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.073282+0000 mon.a (mon.0) 273 : cluster [WRN] RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi032 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpo3z7jlle:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch: docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:32.527 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.073301+0000 mon.a (mon.0) 274 : cluster [WRN] See 'docker run --help'. 2024-04-24T17:29:32.527 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: cluster 2024-04-24T17:29:32.073301+0000 mon.a (mon.0) 274 : cluster [WRN] See 'docker run --help'. 2024-04-24T17:29:32.527 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: audit 2024-04-24T17:29:32.083395+0000 mon.a (mon.0) 275 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:32.527 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: audit 2024-04-24T17:29:32.083395+0000 mon.a (mon.0) 275 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:32.527 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: audit 2024-04-24T17:29:32.089321+0000 mon.a (mon.0) 276 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:32.527 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: audit 2024-04-24T17:29:32.089321+0000 mon.a (mon.0) 276 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:32.527 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: audit 2024-04-24T17:29:32.096440+0000 mon.a (mon.0) 277 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:32.528 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[21457]: audit 2024-04-24T17:29:32.096440+0000 mon.a (mon.0) 277 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:32.528 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:30.990765+0000 mon.a (mon.0) 220 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:32.529 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:30.990765+0000 mon.a (mon.0) 220 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:32.529 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:31.042310+0000 mgr.y (mgr.14152) 62 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:32.529 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:31.042310+0000 mgr.y (mgr.14152) 62 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:32.529 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:31.990982+0000 mon.a (mon.0) 221 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:32.529 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:31.990982+0000 mon.a (mon.0) 221 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:32.529 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.042895+0000 mon.a (mon.0) 222 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-04-24T17:29:32.529 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.042895+0000 mon.a (mon.0) 222 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-04-24T17:29:32.529 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.050928+0000 mon.a (mon.0) 223 : cluster [DBG] monmap epoch 3 2024-04-24T17:29:32.529 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.050928+0000 mon.a (mon.0) 223 : cluster [DBG] monmap epoch 3 2024-04-24T17:29:32.529 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.050983+0000 mon.a (mon.0) 224 : cluster [DBG] fsid a7981b84-025f-11ef-bc93-c7b262605968 2024-04-24T17:29:32.529 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.050983+0000 mon.a (mon.0) 224 : cluster [DBG] fsid a7981b84-025f-11ef-bc93-c7b262605968 2024-04-24T17:29:32.529 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.051029+0000 mon.a (mon.0) 225 : cluster [DBG] last_changed 2024-04-24T17:29:26.991555+0000 2024-04-24T17:29:32.529 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.051029+0000 mon.a (mon.0) 225 : cluster [DBG] last_changed 2024-04-24T17:29:26.991555+0000 2024-04-24T17:29:32.529 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.051052+0000 mon.a (mon.0) 226 : cluster [DBG] created 2024-04-24T17:26:59.605652+0000 2024-04-24T17:29:32.529 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.051052+0000 mon.a (mon.0) 226 : cluster [DBG] created 2024-04-24T17:26:59.605652+0000 2024-04-24T17:29:32.530 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.051076+0000 mon.a (mon.0) 227 : cluster [DBG] min_mon_release 19 (squid) 2024-04-24T17:29:32.530 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.051076+0000 mon.a (mon.0) 227 : cluster [DBG] min_mon_release 19 (squid) 2024-04-24T17:29:32.530 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.051111+0000 mon.a (mon.0) 228 : cluster [DBG] election_strategy: 1 2024-04-24T17:29:32.530 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.051111+0000 mon.a (mon.0) 228 : cluster [DBG] election_strategy: 1 2024-04-24T17:29:32.530 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.051152+0000 mon.a (mon.0) 229 : cluster [DBG] 0: v1:172.21.15.32:6789/0 mon.a 2024-04-24T17:29:32.530 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.051152+0000 mon.a (mon.0) 229 : cluster [DBG] 0: v1:172.21.15.32:6789/0 mon.a 2024-04-24T17:29:32.530 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.051188+0000 mon.a (mon.0) 230 : cluster [DBG] 1: v1:172.21.15.82:6789/0 mon.b 2024-04-24T17:29:32.530 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.051188+0000 mon.a (mon.0) 230 : cluster [DBG] 1: v1:172.21.15.82:6789/0 mon.b 2024-04-24T17:29:32.530 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.051228+0000 mon.a (mon.0) 231 : cluster [DBG] 2: v1:172.21.15.32:6790/0 mon.c 2024-04-24T17:29:32.530 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.051228+0000 mon.a (mon.0) 231 : cluster [DBG] 2: v1:172.21.15.32:6790/0 mon.c 2024-04-24T17:29:32.530 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.071816+0000 mon.a (mon.0) 232 : cluster [DBG] fsmap 2024-04-24T17:29:32.530 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.071816+0000 mon.a (mon.0) 232 : cluster [DBG] fsmap 2024-04-24T17:29:32.530 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.071875+0000 mon.a (mon.0) 233 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-04-24T17:29:32.530 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.071875+0000 mon.a (mon.0) 233 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-04-24T17:29:32.530 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072175+0000 mon.a (mon.0) 234 : cluster [DBG] mgrmap e13: y(active, since 78s) 2024-04-24T17:29:32.531 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072175+0000 mon.a (mon.0) 234 : cluster [DBG] mgrmap e13: y(active, since 78s) 2024-04-24T17:29:32.531 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072457+0000 mon.a (mon.0) 235 : cluster [WRN] Health detail: HEALTH_WARN failed to probe daemons or devices 2024-04-24T17:29:32.531 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072457+0000 mon.a (mon.0) 235 : cluster [WRN] Health detail: HEALTH_WARN failed to probe daemons or devices 2024-04-24T17:29:32.531 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072499+0000 mon.a (mon.0) 236 : cluster [WRN] [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-24T17:29:32.531 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072499+0000 mon.a (mon.0) 236 : cluster [WRN] [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-24T17:29:32.531 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072523+0000 mon.a (mon.0) 237 : cluster [WRN] host smithi082 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T17:29:32.531 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072523+0000 mon.a (mon.0) 237 : cluster [WRN] host smithi082 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T17:29:32.531 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072542+0000 mon.a (mon.0) 238 : cluster [WRN] Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi082 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpbyjftb4d:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch 2024-04-24T17:29:32.531 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072542+0000 mon.a (mon.0) 238 : cluster [WRN] Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi082 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpbyjftb4d:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch 2024-04-24T17:29:32.531 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072562+0000 mon.a (mon.0) 239 : cluster [WRN] /usr/bin/docker: stderr docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:32.531 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072562+0000 mon.a (mon.0) 239 : cluster [WRN] /usr/bin/docker: stderr docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:32.531 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072580+0000 mon.a (mon.0) 240 : cluster [WRN] /usr/bin/docker: stderr See 'docker run --help'. 2024-04-24T17:29:32.531 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072580+0000 mon.a (mon.0) 240 : cluster [WRN] /usr/bin/docker: stderr See 'docker run --help'. 2024-04-24T17:29:32.531 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072599+0000 mon.a (mon.0) 241 : cluster [WRN] Traceback (most recent call last): 2024-04-24T17:29:32.531 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072599+0000 mon.a (mon.0) 241 : cluster [WRN] Traceback (most recent call last): 2024-04-24T17:29:32.531 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072617+0000 mon.a (mon.0) 242 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main 2024-04-24T17:29:32.532 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072617+0000 mon.a (mon.0) 242 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main 2024-04-24T17:29:32.532 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072640+0000 mon.a (mon.0) 243 : cluster [WRN] return _run_code(code, main_globals, None, 2024-04-24T17:29:32.532 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072640+0000 mon.a (mon.0) 243 : cluster [WRN] return _run_code(code, main_globals, None, 2024-04-24T17:29:32.532 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072658+0000 mon.a (mon.0) 244 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 86, in _run_code 2024-04-24T17:29:32.532 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072658+0000 mon.a (mon.0) 244 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 86, in _run_code 2024-04-24T17:29:32.532 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072678+0000 mon.a (mon.0) 245 : cluster [WRN] exec(code, run_globals) 2024-04-24T17:29:32.532 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072678+0000 mon.a (mon.0) 245 : cluster [WRN] exec(code, run_globals) 2024-04-24T17:29:32.532 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072708+0000 mon.a (mon.0) 246 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T17:29:32.532 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072708+0000 mon.a (mon.0) 246 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T17:29:32.532 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072733+0000 mon.a (mon.0) 247 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T17:29:32.532 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072733+0000 mon.a (mon.0) 247 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T17:29:32.532 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072756+0000 mon.a (mon.0) 248 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 408, in _infer_config 2024-04-24T17:29:32.532 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072756+0000 mon.a (mon.0) 248 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 408, in _infer_config 2024-04-24T17:29:32.532 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072778+0000 mon.a (mon.0) 249 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 323, in _infer_fsid 2024-04-24T17:29:32.532 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072778+0000 mon.a (mon.0) 249 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 323, in _infer_fsid 2024-04-24T17:29:32.532 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072805+0000 mon.a (mon.0) 250 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 436, in _infer_image 2024-04-24T17:29:32.533 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072805+0000 mon.a (mon.0) 250 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 436, in _infer_image 2024-04-24T17:29:32.533 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072825+0000 mon.a (mon.0) 251 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 310, in _validate_fsid 2024-04-24T17:29:32.533 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072825+0000 mon.a (mon.0) 251 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 310, in _validate_fsid 2024-04-24T17:29:32.533 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072848+0000 mon.a (mon.0) 252 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3380, in command_ceph_volume 2024-04-24T17:29:32.533 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072848+0000 mon.a (mon.0) 252 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3380, in command_ceph_volume 2024-04-24T17:29:32.533 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072867+0000 mon.a (mon.0) 253 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T17:29:32.533 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072867+0000 mon.a (mon.0) 253 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T17:29:32.533 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072888+0000 mon.a (mon.0) 254 : cluster [WRN] RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi082 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpbyjftb4d:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch: docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:32.533 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072888+0000 mon.a (mon.0) 254 : cluster [WRN] RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi082 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpbyjftb4d:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch: docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:32.533 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072908+0000 mon.a (mon.0) 255 : cluster [WRN] See 'docker run --help'. 2024-04-24T17:29:32.533 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072908+0000 mon.a (mon.0) 255 : cluster [WRN] See 'docker run --help'. 2024-04-24T17:29:32.533 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072926+0000 mon.a (mon.0) 256 : cluster [WRN] host smithi032 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T17:29:32.533 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072926+0000 mon.a (mon.0) 256 : cluster [WRN] host smithi032 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T17:29:32.533 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072947+0000 mon.a (mon.0) 257 : cluster [WRN] Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi032 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpo3z7jlle:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch 2024-04-24T17:29:32.533 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072947+0000 mon.a (mon.0) 257 : cluster [WRN] Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi032 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpo3z7jlle:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch 2024-04-24T17:29:32.534 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072990+0000 mon.a (mon.0) 258 : cluster [WRN] /usr/bin/docker: stderr docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:32.534 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.072990+0000 mon.a (mon.0) 258 : cluster [WRN] /usr/bin/docker: stderr docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:32.534 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.073009+0000 mon.a (mon.0) 259 : cluster [WRN] /usr/bin/docker: stderr See 'docker run --help'. 2024-04-24T17:29:32.534 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.073009+0000 mon.a (mon.0) 259 : cluster [WRN] /usr/bin/docker: stderr See 'docker run --help'. 2024-04-24T17:29:32.534 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.073027+0000 mon.a (mon.0) 260 : cluster [WRN] Traceback (most recent call last): 2024-04-24T17:29:32.534 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.073027+0000 mon.a (mon.0) 260 : cluster [WRN] Traceback (most recent call last): 2024-04-24T17:29:32.534 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.073047+0000 mon.a (mon.0) 261 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main 2024-04-24T17:29:32.534 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.073047+0000 mon.a (mon.0) 261 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main 2024-04-24T17:29:32.534 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.073067+0000 mon.a (mon.0) 262 : cluster [WRN] return _run_code(code, main_globals, None, 2024-04-24T17:29:32.534 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.073067+0000 mon.a (mon.0) 262 : cluster [WRN] return _run_code(code, main_globals, None, 2024-04-24T17:29:32.534 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.073086+0000 mon.a (mon.0) 263 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 86, in _run_code 2024-04-24T17:29:32.534 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.073086+0000 mon.a (mon.0) 263 : cluster [WRN] File "/usr/lib/python3.10/runpy.py", line 86, in _run_code 2024-04-24T17:29:32.534 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.073104+0000 mon.a (mon.0) 264 : cluster [WRN] exec(code, run_globals) 2024-04-24T17:29:32.534 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.073104+0000 mon.a (mon.0) 264 : cluster [WRN] exec(code, run_globals) 2024-04-24T17:29:32.534 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.073126+0000 mon.a (mon.0) 265 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T17:29:32.535 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.073126+0000 mon.a (mon.0) 265 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5642, in 2024-04-24T17:29:32.535 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.073144+0000 mon.a (mon.0) 266 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T17:29:32.535 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.073144+0000 mon.a (mon.0) 266 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 5630, in main 2024-04-24T17:29:32.535 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.073165+0000 mon.a (mon.0) 267 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 408, in _infer_config 2024-04-24T17:29:32.535 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.073165+0000 mon.a (mon.0) 267 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 408, in _infer_config 2024-04-24T17:29:32.535 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.073183+0000 mon.a (mon.0) 268 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 323, in _infer_fsid 2024-04-24T17:29:32.535 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.073183+0000 mon.a (mon.0) 268 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 323, in _infer_fsid 2024-04-24T17:29:32.535 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.073201+0000 mon.a (mon.0) 269 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 436, in _infer_image 2024-04-24T17:29:32.535 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.073201+0000 mon.a (mon.0) 269 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 436, in _infer_image 2024-04-24T17:29:32.535 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.073221+0000 mon.a (mon.0) 270 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 310, in _validate_fsid 2024-04-24T17:29:32.535 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.073221+0000 mon.a (mon.0) 270 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 310, in _validate_fsid 2024-04-24T17:29:32.535 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.073241+0000 mon.a (mon.0) 271 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3380, in command_ceph_volume 2024-04-24T17:29:32.535 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.073241+0000 mon.a (mon.0) 271 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/__main__.py", line 3380, in command_ceph_volume 2024-04-24T17:29:32.535 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.073259+0000 mon.a (mon.0) 272 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T17:29:32.535 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.073259+0000 mon.a (mon.0) 272 : cluster [WRN] File "/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/cephadm.1f08c70ed1be73eb5430d661f765904c2335d752edef06817644566059cdce9e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T17:29:32.536 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.073282+0000 mon.a (mon.0) 273 : cluster [WRN] RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi032 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpo3z7jlle:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch: docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:32.536 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.073282+0000 mon.a (mon.0) 273 : cluster [WRN] RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a -e NODE_NAME=smithi032 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpo3z7jlle:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7d58b69a27db4d75e45a15a3a40c6148076eda7e4bc8fdbe5fb876f29532e51a inventory --format=json-pretty --filter-for-batch: docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:32.536 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.073301+0000 mon.a (mon.0) 274 : cluster [WRN] See 'docker run --help'. 2024-04-24T17:29:32.536 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: cluster 2024-04-24T17:29:32.073301+0000 mon.a (mon.0) 274 : cluster [WRN] See 'docker run --help'. 2024-04-24T17:29:32.536 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:32.083395+0000 mon.a (mon.0) 275 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:32.536 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:32.083395+0000 mon.a (mon.0) 275 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:32.536 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:32.089321+0000 mon.a (mon.0) 276 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:32.536 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:32.089321+0000 mon.a (mon.0) 276 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:32.536 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:32.096440+0000 mon.a (mon.0) 277 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:32.536 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:32 smithi032 bash[27083]: audit 2024-04-24T17:29:32.096440+0000 mon.a (mon.0) 277 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:32.790 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-24T17:29:32.790 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a7981b84-025f-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-24T17:29:33.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:33 smithi032 bash[21457]: audit 2024-04-24T17:29:32.990324+0000 mon.a (mon.0) 278 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:33.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:33 smithi032 bash[21457]: audit 2024-04-24T17:29:32.990324+0000 mon.a (mon.0) 278 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:33.503 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:33 smithi032 bash[27083]: audit 2024-04-24T17:29:32.990324+0000 mon.a (mon.0) 278 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:33.503 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:33 smithi032 bash[27083]: audit 2024-04-24T17:29:32.990324+0000 mon.a (mon.0) 278 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:33.630 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:33 smithi082 bash[21793]: audit 2024-04-24T17:29:32.990324+0000 mon.a (mon.0) 278 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:33.630 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:33 smithi082 bash[21793]: audit 2024-04-24T17:29:32.990324+0000 mon.a (mon.0) 278 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T17:29:34.252 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:29:33 smithi032 bash[21712]: debug 2024-04-24T17:29:33.990+0000 7fd3d5282700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-24T17:29:34.252 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:34 smithi032 bash[27083]: cluster 2024-04-24T17:29:33.042820+0000 mgr.y (mgr.14152) 63 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:34.253 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:34 smithi032 bash[27083]: cluster 2024-04-24T17:29:33.042820+0000 mgr.y (mgr.14152) 63 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:34.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:34 smithi032 bash[21457]: cluster 2024-04-24T17:29:33.042820+0000 mgr.y (mgr.14152) 63 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:34.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:34 smithi032 bash[21457]: cluster 2024-04-24T17:29:33.042820+0000 mgr.y (mgr.14152) 63 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:34.630 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:34 smithi082 bash[21793]: cluster 2024-04-24T17:29:33.042820+0000 mgr.y (mgr.14152) 63 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:34.630 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:34 smithi082 bash[21793]: cluster 2024-04-24T17:29:33.042820+0000 mgr.y (mgr.14152) 63 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:36.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:36 smithi082 bash[21793]: cluster 2024-04-24T17:29:35.043738+0000 mgr.y (mgr.14152) 64 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:36.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:36 smithi082 bash[21793]: cluster 2024-04-24T17:29:35.043738+0000 mgr.y (mgr.14152) 64 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:36.502 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:36 smithi032 bash[27083]: cluster 2024-04-24T17:29:35.043738+0000 mgr.y (mgr.14152) 64 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:36.502 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:36 smithi032 bash[27083]: cluster 2024-04-24T17:29:35.043738+0000 mgr.y (mgr.14152) 64 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:36.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:36 smithi032 bash[21457]: cluster 2024-04-24T17:29:35.043738+0000 mgr.y (mgr.14152) 64 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:36.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:36 smithi032 bash[21457]: cluster 2024-04-24T17:29:35.043738+0000 mgr.y (mgr.14152) 64 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:37.527 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T17:29:38.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:38 smithi032 bash[21457]: cluster 2024-04-24T17:29:37.044702+0000 mgr.y (mgr.14152) 65 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:38.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:38 smithi032 bash[21457]: cluster 2024-04-24T17:29:37.044702+0000 mgr.y (mgr.14152) 65 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:38.502 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:38 smithi032 bash[27083]: cluster 2024-04-24T17:29:37.044702+0000 mgr.y (mgr.14152) 65 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:38.502 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:38 smithi032 bash[27083]: cluster 2024-04-24T17:29:37.044702+0000 mgr.y (mgr.14152) 65 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:38.630 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:38 smithi082 bash[21793]: cluster 2024-04-24T17:29:37.044702+0000 mgr.y (mgr.14152) 65 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:38.630 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:38 smithi082 bash[21793]: cluster 2024-04-24T17:29:37.044702+0000 mgr.y (mgr.14152) 65 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:38.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.793+0000 7f963ef95700 1 -- 172.21.15.32:0/1843835091 >> v1:172.21.15.32:6789/0 conn(0x7f9638101ea0 legacy=0x7f9638102280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:29:38.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.793+0000 7f963ef95700 1 -- 172.21.15.32:0/1843835091 shutdown_connections 2024-04-24T17:29:38.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.793+0000 7f963ef95700 1 -- 172.21.15.32:0/1843835091 >> 172.21.15.32:0/1843835091 conn(0x7f96380fd730 msgr2=0x7f96380ffb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:29:38.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.793+0000 7f963ef95700 1 -- 172.21.15.32:0/1843835091 shutdown_connections 2024-04-24T17:29:38.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.793+0000 7f963ef95700 1 -- 172.21.15.32:0/1843835091 wait complete. 2024-04-24T17:29:38.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.793+0000 7f963ef95700 1 librados: starting msgr at 2024-04-24T17:29:38.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.793+0000 7f963ef95700 1 librados: starting objecter 2024-04-24T17:29:38.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.793+0000 7f963ef95700 1 Processor -- start 2024-04-24T17:29:38.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.793+0000 7f963ef95700 1 -- start start 2024-04-24T17:29:38.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.793+0000 7f963ef95700 1 librados: setting wanted keys 2024-04-24T17:29:38.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.793+0000 7f963ef95700 1 librados: calling monclient init 2024-04-24T17:29:38.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.793+0000 7f963ef95700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f96381a9e20 con 0x7f96381a6150 2024-04-24T17:29:38.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.793+0000 7f963ef95700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f96381ab040 con 0x7f963819a620 2024-04-24T17:29:38.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.793+0000 7f963ef95700 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f96381ac280 con 0x7f9638101ea0 2024-04-24T17:29:38.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.793+0000 7f963cd31700 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7f9638101ea0 0x7f9638199f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:50804/0 (socket says 172.21.15.32:50804) 2024-04-24T17:29:38.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.793+0000 7f963cd31700 1 -- 172.21.15.32:0/3754769807 learned_addr learned my addr 172.21.15.32:0/3754769807 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-04-24T17:29:38.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.793+0000 7f9635ffb700 10 client.?.objecter ms_handle_connect 0x7f9638101ea0 2024-04-24T17:29:38.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.793+0000 7f9635ffb700 10 client.?.objecter resend_mon_ops 2024-04-24T17:29:38.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.793+0000 7f9635ffb700 10 client.?.objecter ms_handle_connect 0x7f963819a620 2024-04-24T17:29:38.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.793+0000 7f9635ffb700 10 client.?.objecter resend_mon_ops 2024-04-24T17:29:38.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.793+0000 7f9635ffb700 1 -- 172.21.15.32:0/3754769807 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4107826374 0 0) 0x7f96381ac280 con 0x7f9638101ea0 2024-04-24T17:29:38.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.793+0000 7f9635ffb700 1 -- 172.21.15.32:0/3754769807 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9620003bf0 con 0x7f9638101ea0 2024-04-24T17:29:38.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.797+0000 7f9635ffb700 1 -- 172.21.15.32:0/3754769807 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2341757322 0 0) 0x7f96381ab040 con 0x7f963819a620 2024-04-24T17:29:38.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.797+0000 7f9635ffb700 1 -- 172.21.15.32:0/3754769807 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96381ac280 con 0x7f963819a620 2024-04-24T17:29:38.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.797+0000 7f9635ffb700 1 -- 172.21.15.32:0/3754769807 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3017485802 0 0) 0x7f9620003bf0 con 0x7f9638101ea0 2024-04-24T17:29:38.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.797+0000 7f9635ffb700 1 -- 172.21.15.32:0/3754769807 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f96381ab040 con 0x7f9638101ea0 2024-04-24T17:29:38.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.797+0000 7f9635ffb700 1 -- 172.21.15.32:0/3754769807 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 512931633 0 0) 0x7f9628001ae0 con 0x7f9638101ea0 2024-04-24T17:29:38.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.797+0000 7f9635ffb700 10 client.?.objecter ms_handle_connect 0x7f96381a6150 2024-04-24T17:29:38.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.797+0000 7f9635ffb700 10 client.?.objecter resend_mon_ops 2024-04-24T17:29:38.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.797+0000 7f9635ffb700 1 -- 172.21.15.32:0/3754769807 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3392174983 0 0) 0x7f96381ac280 con 0x7f963819a620 2024-04-24T17:29:38.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.797+0000 7f9635ffb700 1 -- 172.21.15.32:0/3754769807 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9620003bf0 con 0x7f963819a620 2024-04-24T17:29:38.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.797+0000 7f9635ffb700 1 -- 172.21.15.32:0/3754769807 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2556692894 0 0) 0x7f96381a9e20 con 0x7f96381a6150 2024-04-24T17:29:38.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.797+0000 7f9635ffb700 1 -- 172.21.15.32:0/3754769807 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96381ac280 con 0x7f96381a6150 2024-04-24T17:29:38.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.797+0000 7f9635ffb700 1 -- 172.21.15.32:0/3754769807 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 512931633 0 0) 0x7f962c001830 con 0x7f963819a620 2024-04-24T17:29:38.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.797+0000 7f9635ffb700 1 -- 172.21.15.32:0/3754769807 <== mon.2 v1:172.21.15.32:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3639299805 0 0) 0x7f96381ab040 con 0x7f9638101ea0 2024-04-24T17:29:38.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.797+0000 7f9635ffb700 1 -- 172.21.15.32:0/3754769807 >> v1:172.21.15.82:6789/0 conn(0x7f963819a620 legacy=0x7f96381a4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:29:38.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.797+0000 7f9635ffb700 1 -- 172.21.15.32:0/3754769807 >> v1:172.21.15.32:6789/0 conn(0x7f96381a6150 legacy=0x7f96381a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:29:38.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.797+0000 7f9635ffb700 1 -- 172.21.15.32:0/3754769807 --> v1:172.21.15.32:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96381ad4c0 con 0x7f9638101ea0 2024-04-24T17:29:38.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.797+0000 7f963ef95700 1 -- 172.21.15.32:0/3754769807 --> v1:172.21.15.32:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f96381ab270 con 0x7f9638101ea0 2024-04-24T17:29:38.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.797+0000 7f963ef95700 10 client.24101.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-24T17:29:38.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.797+0000 7f963ef95700 1 -- 172.21.15.32:0/3754769807 --> v1:172.21.15.32:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f96381ab6d0 con 0x7f9638101ea0 2024-04-24T17:29:38.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.797+0000 7f963ef95700 1 librados: init done 2024-04-24T17:29:38.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.797+0000 7f9635ffb700 1 -- 172.21.15.32:0/3754769807 <== mon.2 v1:172.21.15.32:6790/0 5 ==== config(25 keys) ==== 1042+0+0 (unknown 2768897130 0 0) 0x7f96280011b0 con 0x7f9638101ea0 2024-04-24T17:29:38.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.797+0000 7f9635ffb700 1 -- 172.21.15.32:0/3754769807 <== mon.2 v1:172.21.15.32:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 512931633 0 0) 0x7f9628004df0 con 0x7f9638101ea0 2024-04-24T17:29:38.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.797+0000 7f9635ffb700 1 -- 172.21.15.32:0/3754769807 <== mon.2 v1:172.21.15.32:6790/0 7 ==== mgrmap(e 13) ==== 46579+0+0 (unknown 583022566 0 0) 0x7f9628004fb0 con 0x7f9638101ea0 2024-04-24T17:29:38.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.797+0000 7f963ef95700 1 -- 172.21.15.32:0/3754769807 --> v1:172.21.15.32:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9624005120 con 0x7f9638101ea0 2024-04-24T17:29:38.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.797+0000 7f9635ffb700 1 -- 172.21.15.32:0/3754769807 <== mon.2 v1:172.21.15.32:6790/0 8 ==== osd_map(4..4 src has 1..4) ==== 1069+0+0 (unknown 1280957937 0 0) 0x7f9628048ef0 con 0x7f9638101ea0 2024-04-24T17:29:38.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.797+0000 7f9635ffb700 10 client.24101.objecter ms_dispatch 0x7f9638001550 osd_map(4..4 src has 1..4) 2024-04-24T17:29:38.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.797+0000 7f9635ffb700 3 client.24101.objecter handle_osd_map got epochs [4,4] > 0 2024-04-24T17:29:38.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.797+0000 7f9635ffb700 3 client.24101.objecter handle_osd_map decoding full epoch 4 2024-04-24T17:29:38.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.797+0000 7f9635ffb700 20 client.24101.objecter dump_active .. 0 homeless 2024-04-24T17:29:38.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.797+0000 7f9635ffb700 10 client.24101.objecter ms_handle_connect 0x7f962003c350 2024-04-24T17:29:38.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:38.801+0000 7f9635ffb700 1 -- 172.21.15.32:0/3754769807 <== mon.2 v1:172.21.15.32:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188728 (unknown 1092875540 0 2656817166) 0x7f96280176a0 con 0x7f9638101ea0 2024-04-24T17:29:39.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:39.017+0000 7f963ef95700 1 -- 172.21.15.32:0/3754769807 --> v1:172.21.15.32:6790/0 -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f9624005410 con 0x7f9638101ea0 2024-04-24T17:29:39.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:39.021+0000 7f9635ffb700 1 -- 172.21.15.32:0/3754769807 <== mon.2 v1:172.21.15.32:6790/0 10 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+190 (unknown 2189217731 0 199378286) 0x7f9628016fa0 con 0x7f9638101ea0 2024-04-24T17:29:39.022 INFO:teuthology.orchestra.run.smithi032.stdout:# minimal ceph.conf for a7981b84-025f-11ef-bc93-c7b262605968 2024-04-24T17:29:39.022 INFO:teuthology.orchestra.run.smithi032.stdout:[global] 2024-04-24T17:29:39.022 INFO:teuthology.orchestra.run.smithi032.stdout: fsid = a7981b84-025f-11ef-bc93-c7b262605968 2024-04-24T17:29:39.022 INFO:teuthology.orchestra.run.smithi032.stdout: mon_host = 172.21.15.32:6789/0 172.21.15.82:6789/0 v1:172.21.15.32:6790/0 2024-04-24T17:29:39.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:39.021+0000 7f963ef95700 10 librados: watch_flush enter 2024-04-24T17:29:39.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:39.021+0000 7f963ef95700 10 librados: watch_flush exit 2024-04-24T17:29:39.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:39.021+0000 7f963ef95700 20 client.24101.objecter shutdown clearing up homeless session... 2024-04-24T17:29:39.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:39.021+0000 7f963ef95700 10 client.24101.objecter successfully canceled tick 2024-04-24T17:29:39.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:39.021+0000 7f963ef95700 1 -- 172.21.15.32:0/3754769807 >> v1:172.21.15.32:6800/1847389373 conn(0x7f962003c350 legacy=0x7f962003e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:29:39.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:39.021+0000 7f963ef95700 1 -- 172.21.15.32:0/3754769807 >> v1:172.21.15.32:6790/0 conn(0x7f9638101ea0 legacy=0x7f9638199f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:29:39.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:39.021+0000 7f963ef95700 1 -- 172.21.15.32:0/3754769807 shutdown_connections 2024-04-24T17:29:39.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:39.021+0000 7f963ef95700 1 -- 172.21.15.32:0/3754769807 >> 172.21.15.32:0/3754769807 conn(0x7f96380fd730 msgr2=0x7f963806b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:29:39.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:39.021+0000 7f963ef95700 1 -- 172.21.15.32:0/3754769807 shutdown_connections 2024-04-24T17:29:39.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:39.021+0000 7f963ef95700 1 -- 172.21.15.32:0/3754769807 wait complete. 2024-04-24T17:29:39.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-24T17:29:39.021+0000 7f963ef95700 1 librados: shutdown 2024-04-24T17:29:39.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:39 smithi032 bash[21457]: audit 2024-04-24T17:29:39.023186+0000 mon.c (mon.2) 2 : audit [DBG] from='client.? v1:172.21.15.32:0/3754769807' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:39.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:39 smithi032 bash[21457]: audit 2024-04-24T17:29:39.023186+0000 mon.c (mon.2) 2 : audit [DBG] from='client.? v1:172.21.15.32:0/3754769807' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:39.502 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:39 smithi032 bash[27083]: audit 2024-04-24T17:29:39.023186+0000 mon.c (mon.2) 2 : audit [DBG] from='client.? v1:172.21.15.32:0/3754769807' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:39.502 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:39 smithi032 bash[27083]: audit 2024-04-24T17:29:39.023186+0000 mon.c (mon.2) 2 : audit [DBG] from='client.? v1:172.21.15.32:0/3754769807' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:39.630 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:39 smithi082 bash[21793]: audit 2024-04-24T17:29:39.023186+0000 mon.c (mon.2) 2 : audit [DBG] from='client.? v1:172.21.15.32:0/3754769807' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:39.630 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:39 smithi082 bash[21793]: audit 2024-04-24T17:29:39.023186+0000 mon.c (mon.2) 2 : audit [DBG] from='client.? v1:172.21.15.32:0/3754769807' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:40.049 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-24T17:29:40.049 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-24T17:29:40.049 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/ceph/ceph.conf 2024-04-24T17:29:40.063 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-24T17:29:40.063 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-24T17:29:40.118 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-04-24T17:29:40.118 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/ceph/ceph.conf 2024-04-24T17:29:40.131 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-04-24T17:29:40.132 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-24T17:29:40.188 INFO:tasks.cephadm:Adding mgr.y on smithi032 2024-04-24T17:29:40.188 INFO:tasks.cephadm:Adding mgr.x on smithi082 2024-04-24T17:29:40.188 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a7981b84-025f-11ef-bc93-c7b262605968 -- ceph orch apply mgr '2;smithi032=y;smithi082=x' 2024-04-24T17:29:40.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:40 smithi082 bash[21793]: cluster 2024-04-24T17:29:39.045137+0000 mgr.y (mgr.14152) 66 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:40.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:40 smithi082 bash[21793]: cluster 2024-04-24T17:29:39.045137+0000 mgr.y (mgr.14152) 66 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:40.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:40 smithi082 bash[21793]: audit 2024-04-24T17:29:40.054006+0000 mon.a (mon.0) 279 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:40.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:40 smithi082 bash[21793]: audit 2024-04-24T17:29:40.054006+0000 mon.a (mon.0) 279 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:40.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:40 smithi082 bash[21793]: audit 2024-04-24T17:29:40.055476+0000 mon.a (mon.0) 280 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T17:29:40.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:40 smithi082 bash[21793]: audit 2024-04-24T17:29:40.055476+0000 mon.a (mon.0) 280 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T17:29:40.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:40 smithi032 bash[21457]: cluster 2024-04-24T17:29:39.045137+0000 mgr.y (mgr.14152) 66 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:40.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:40 smithi032 bash[21457]: cluster 2024-04-24T17:29:39.045137+0000 mgr.y (mgr.14152) 66 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:40.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:40 smithi032 bash[21457]: audit 2024-04-24T17:29:40.054006+0000 mon.a (mon.0) 279 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:40.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:40 smithi032 bash[21457]: audit 2024-04-24T17:29:40.054006+0000 mon.a (mon.0) 279 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:40.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:40 smithi032 bash[21457]: audit 2024-04-24T17:29:40.055476+0000 mon.a (mon.0) 280 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T17:29:40.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:40 smithi032 bash[21457]: audit 2024-04-24T17:29:40.055476+0000 mon.a (mon.0) 280 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T17:29:40.503 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:40 smithi032 bash[27083]: cluster 2024-04-24T17:29:39.045137+0000 mgr.y (mgr.14152) 66 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:40.503 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:40 smithi032 bash[27083]: cluster 2024-04-24T17:29:39.045137+0000 mgr.y (mgr.14152) 66 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:40.503 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:40 smithi032 bash[27083]: audit 2024-04-24T17:29:40.054006+0000 mon.a (mon.0) 279 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:40.503 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:40 smithi032 bash[27083]: audit 2024-04-24T17:29:40.054006+0000 mon.a (mon.0) 279 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:40.503 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:40 smithi032 bash[27083]: audit 2024-04-24T17:29:40.055476+0000 mon.a (mon.0) 280 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T17:29:40.503 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:40 smithi032 bash[27083]: audit 2024-04-24T17:29:40.055476+0000 mon.a (mon.0) 280 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T17:29:41.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[21457]: cephadm 2024-04-24T17:29:40.057296+0000 mgr.y (mgr.14152) 67 : cephadm [INF] Updating smithi032:/etc/ceph/ceph.conf 2024-04-24T17:29:41.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[21457]: cephadm 2024-04-24T17:29:40.057296+0000 mgr.y (mgr.14152) 67 : cephadm [INF] Updating smithi032:/etc/ceph/ceph.conf 2024-04-24T17:29:41.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[21457]: cephadm 2024-04-24T17:29:40.057716+0000 mgr.y (mgr.14152) 68 : cephadm [INF] Updating smithi082:/etc/ceph/ceph.conf 2024-04-24T17:29:41.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[21457]: cephadm 2024-04-24T17:29:40.057716+0000 mgr.y (mgr.14152) 68 : cephadm [INF] Updating smithi082:/etc/ceph/ceph.conf 2024-04-24T17:29:41.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[21457]: cephadm 2024-04-24T17:29:40.222327+0000 mgr.y (mgr.14152) 69 : cephadm [INF] Updating smithi032:/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T17:29:41.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[21457]: cephadm 2024-04-24T17:29:40.222327+0000 mgr.y (mgr.14152) 69 : cephadm [INF] Updating smithi032:/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T17:29:41.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[21457]: cephadm 2024-04-24T17:29:40.222842+0000 mgr.y (mgr.14152) 70 : cephadm [INF] Updating smithi082:/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T17:29:41.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[21457]: cephadm 2024-04-24T17:29:40.222842+0000 mgr.y (mgr.14152) 70 : cephadm [INF] Updating smithi082:/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T17:29:41.503 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[21457]: audit 2024-04-24T17:29:40.671611+0000 mon.a (mon.0) 281 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:41.504 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[21457]: audit 2024-04-24T17:29:40.671611+0000 mon.a (mon.0) 281 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:41.504 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[21457]: audit 2024-04-24T17:29:40.675405+0000 mon.a (mon.0) 282 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:41.504 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[21457]: audit 2024-04-24T17:29:40.675405+0000 mon.a (mon.0) 282 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:41.504 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[21457]: audit 2024-04-24T17:29:40.680452+0000 mon.a (mon.0) 283 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:41.504 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[21457]: audit 2024-04-24T17:29:40.680452+0000 mon.a (mon.0) 283 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:41.504 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[21457]: audit 2024-04-24T17:29:40.721688+0000 mon.a (mon.0) 284 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:41.504 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[21457]: audit 2024-04-24T17:29:40.721688+0000 mon.a (mon.0) 284 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:41.504 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[21457]: audit 2024-04-24T17:29:40.726038+0000 mon.a (mon.0) 285 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:41.504 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[21457]: audit 2024-04-24T17:29:40.726038+0000 mon.a (mon.0) 285 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:41.504 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[21457]: audit 2024-04-24T17:29:40.730166+0000 mon.a (mon.0) 286 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:41.504 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[21457]: audit 2024-04-24T17:29:40.730166+0000 mon.a (mon.0) 286 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:41.504 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[21457]: audit 2024-04-24T17:29:40.734373+0000 mon.a (mon.0) 287 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:41.504 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[21457]: audit 2024-04-24T17:29:40.734373+0000 mon.a (mon.0) 287 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:41.504 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[21457]: audit 2024-04-24T17:29:40.735069+0000 mon.a (mon.0) 288 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T17:29:41.505 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[27083]: cephadm 2024-04-24T17:29:40.057296+0000 mgr.y (mgr.14152) 67 : cephadm [INF] Updating smithi032:/etc/ceph/ceph.conf 2024-04-24T17:29:41.505 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[27083]: cephadm 2024-04-24T17:29:40.057296+0000 mgr.y (mgr.14152) 67 : cephadm [INF] Updating smithi032:/etc/ceph/ceph.conf 2024-04-24T17:29:41.505 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[27083]: cephadm 2024-04-24T17:29:40.057716+0000 mgr.y (mgr.14152) 68 : cephadm [INF] Updating smithi082:/etc/ceph/ceph.conf 2024-04-24T17:29:41.505 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[27083]: cephadm 2024-04-24T17:29:40.057716+0000 mgr.y (mgr.14152) 68 : cephadm [INF] Updating smithi082:/etc/ceph/ceph.conf 2024-04-24T17:29:41.505 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[27083]: cephadm 2024-04-24T17:29:40.222327+0000 mgr.y (mgr.14152) 69 : cephadm [INF] Updating smithi032:/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T17:29:41.505 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[27083]: cephadm 2024-04-24T17:29:40.222327+0000 mgr.y (mgr.14152) 69 : cephadm [INF] Updating smithi032:/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T17:29:41.505 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[27083]: cephadm 2024-04-24T17:29:40.222842+0000 mgr.y (mgr.14152) 70 : cephadm [INF] Updating smithi082:/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T17:29:41.505 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[27083]: cephadm 2024-04-24T17:29:40.222842+0000 mgr.y (mgr.14152) 70 : cephadm [INF] Updating smithi082:/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T17:29:41.505 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[27083]: audit 2024-04-24T17:29:40.671611+0000 mon.a (mon.0) 281 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:41.505 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[27083]: audit 2024-04-24T17:29:40.671611+0000 mon.a (mon.0) 281 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:41.505 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[27083]: audit 2024-04-24T17:29:40.675405+0000 mon.a (mon.0) 282 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:41.505 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[27083]: audit 2024-04-24T17:29:40.675405+0000 mon.a (mon.0) 282 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:41.505 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[27083]: audit 2024-04-24T17:29:40.680452+0000 mon.a (mon.0) 283 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:41.506 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[27083]: audit 2024-04-24T17:29:40.680452+0000 mon.a (mon.0) 283 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:41.506 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[27083]: audit 2024-04-24T17:29:40.721688+0000 mon.a (mon.0) 284 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:41.506 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[27083]: audit 2024-04-24T17:29:40.721688+0000 mon.a (mon.0) 284 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:41.506 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[27083]: audit 2024-04-24T17:29:40.726038+0000 mon.a (mon.0) 285 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:41.506 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[27083]: audit 2024-04-24T17:29:40.726038+0000 mon.a (mon.0) 285 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:41.506 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[27083]: audit 2024-04-24T17:29:40.730166+0000 mon.a (mon.0) 286 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:41.506 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[27083]: audit 2024-04-24T17:29:40.730166+0000 mon.a (mon.0) 286 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:41.506 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[27083]: audit 2024-04-24T17:29:40.734373+0000 mon.a (mon.0) 287 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:41.506 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[27083]: audit 2024-04-24T17:29:40.734373+0000 mon.a (mon.0) 287 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:41.506 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[27083]: audit 2024-04-24T17:29:40.735069+0000 mon.a (mon.0) 288 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T17:29:41.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[21457]: audit 2024-04-24T17:29:40.735069+0000 mon.a (mon.0) 288 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T17:29:41.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[21457]: audit 2024-04-24T17:29:40.736039+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T17:29:41.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[21457]: audit 2024-04-24T17:29:40.736039+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T17:29:41.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[21457]: audit 2024-04-24T17:29:40.737016+0000 mon.a (mon.0) 290 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:41.507 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[21457]: audit 2024-04-24T17:29:40.737016+0000 mon.a (mon.0) 290 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:41.507 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[27083]: audit 2024-04-24T17:29:40.735069+0000 mon.a (mon.0) 288 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T17:29:41.507 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[27083]: audit 2024-04-24T17:29:40.736039+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T17:29:41.507 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[27083]: audit 2024-04-24T17:29:40.736039+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T17:29:41.507 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[27083]: audit 2024-04-24T17:29:40.737016+0000 mon.a (mon.0) 290 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:41.507 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:41 smithi032 bash[27083]: audit 2024-04-24T17:29:40.737016+0000 mon.a (mon.0) 290 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:41.630 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:41 smithi082 bash[21793]: cephadm 2024-04-24T17:29:40.057296+0000 mgr.y (mgr.14152) 67 : cephadm [INF] Updating smithi032:/etc/ceph/ceph.conf 2024-04-24T17:29:41.631 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:41 smithi082 bash[21793]: cephadm 2024-04-24T17:29:40.057296+0000 mgr.y (mgr.14152) 67 : cephadm [INF] Updating smithi032:/etc/ceph/ceph.conf 2024-04-24T17:29:41.631 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:41 smithi082 bash[21793]: cephadm 2024-04-24T17:29:40.057716+0000 mgr.y (mgr.14152) 68 : cephadm [INF] Updating smithi082:/etc/ceph/ceph.conf 2024-04-24T17:29:41.631 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:41 smithi082 bash[21793]: cephadm 2024-04-24T17:29:40.057716+0000 mgr.y (mgr.14152) 68 : cephadm [INF] Updating smithi082:/etc/ceph/ceph.conf 2024-04-24T17:29:41.631 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:41 smithi082 bash[21793]: cephadm 2024-04-24T17:29:40.222327+0000 mgr.y (mgr.14152) 69 : cephadm [INF] Updating smithi032:/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T17:29:41.631 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:41 smithi082 bash[21793]: cephadm 2024-04-24T17:29:40.222327+0000 mgr.y (mgr.14152) 69 : cephadm [INF] Updating smithi032:/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T17:29:41.631 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:41 smithi082 bash[21793]: cephadm 2024-04-24T17:29:40.222842+0000 mgr.y (mgr.14152) 70 : cephadm [INF] Updating smithi082:/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T17:29:41.631 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:41 smithi082 bash[21793]: cephadm 2024-04-24T17:29:40.222842+0000 mgr.y (mgr.14152) 70 : cephadm [INF] Updating smithi082:/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T17:29:41.631 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:41 smithi082 bash[21793]: audit 2024-04-24T17:29:40.671611+0000 mon.a (mon.0) 281 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:41.631 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:41 smithi082 bash[21793]: audit 2024-04-24T17:29:40.671611+0000 mon.a (mon.0) 281 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:41.631 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:41 smithi082 bash[21793]: audit 2024-04-24T17:29:40.675405+0000 mon.a (mon.0) 282 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:41.631 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:41 smithi082 bash[21793]: audit 2024-04-24T17:29:40.675405+0000 mon.a (mon.0) 282 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:41.631 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:41 smithi082 bash[21793]: audit 2024-04-24T17:29:40.680452+0000 mon.a (mon.0) 283 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:41.631 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:41 smithi082 bash[21793]: audit 2024-04-24T17:29:40.680452+0000 mon.a (mon.0) 283 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:41.631 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:41 smithi082 bash[21793]: audit 2024-04-24T17:29:40.721688+0000 mon.a (mon.0) 284 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:41.632 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:41 smithi082 bash[21793]: audit 2024-04-24T17:29:40.721688+0000 mon.a (mon.0) 284 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:41.632 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:41 smithi082 bash[21793]: audit 2024-04-24T17:29:40.726038+0000 mon.a (mon.0) 285 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:41.632 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:41 smithi082 bash[21793]: audit 2024-04-24T17:29:40.726038+0000 mon.a (mon.0) 285 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:41.632 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:41 smithi082 bash[21793]: audit 2024-04-24T17:29:40.730166+0000 mon.a (mon.0) 286 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:41.632 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:41 smithi082 bash[21793]: audit 2024-04-24T17:29:40.730166+0000 mon.a (mon.0) 286 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:41.632 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:41 smithi082 bash[21793]: audit 2024-04-24T17:29:40.734373+0000 mon.a (mon.0) 287 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:41.632 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:41 smithi082 bash[21793]: audit 2024-04-24T17:29:40.734373+0000 mon.a (mon.0) 287 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:41.632 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:41 smithi082 bash[21793]: audit 2024-04-24T17:29:40.735069+0000 mon.a (mon.0) 288 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T17:29:41.632 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:41 smithi082 bash[21793]: audit 2024-04-24T17:29:40.735069+0000 mon.a (mon.0) 288 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T17:29:41.632 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:41 smithi082 bash[21793]: audit 2024-04-24T17:29:40.736039+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T17:29:41.632 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:41 smithi082 bash[21793]: audit 2024-04-24T17:29:40.736039+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T17:29:41.632 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:41 smithi082 bash[21793]: audit 2024-04-24T17:29:40.737016+0000 mon.a (mon.0) 290 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:41.632 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:41 smithi082 bash[21793]: audit 2024-04-24T17:29:40.737016+0000 mon.a (mon.0) 290 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:42.436 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:42 smithi032 bash[27083]: cluster 2024-04-24T17:29:40.676226+0000 mgr.y (mgr.14152) 71 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:42.436 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:42 smithi032 bash[27083]: cluster 2024-04-24T17:29:40.676226+0000 mgr.y (mgr.14152) 71 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:42.436 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:42 smithi032 bash[27083]: cluster 2024-04-24T17:29:40.677171+0000 mgr.y (mgr.14152) 72 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:42.436 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:42 smithi032 bash[27083]: cluster 2024-04-24T17:29:40.677171+0000 mgr.y (mgr.14152) 72 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:42.436 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:42 smithi032 bash[27083]: cephadm 2024-04-24T17:29:40.734757+0000 mgr.y (mgr.14152) 73 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2024-04-24T17:29:42.436 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:42 smithi032 bash[27083]: cephadm 2024-04-24T17:29:40.734757+0000 mgr.y (mgr.14152) 73 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2024-04-24T17:29:42.436 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:42 smithi032 bash[27083]: cephadm 2024-04-24T17:29:40.738182+0000 mgr.y (mgr.14152) 74 : cephadm [INF] Reconfiguring daemon mon.a on smithi032 2024-04-24T17:29:42.436 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:42 smithi032 bash[27083]: cephadm 2024-04-24T17:29:40.738182+0000 mgr.y (mgr.14152) 74 : cephadm [INF] Reconfiguring daemon mon.a on smithi032 2024-04-24T17:29:42.436 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:42 smithi032 bash[21457]: cluster 2024-04-24T17:29:40.676226+0000 mgr.y (mgr.14152) 71 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:42.436 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:42 smithi032 bash[21457]: cluster 2024-04-24T17:29:40.676226+0000 mgr.y (mgr.14152) 71 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:42.437 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:42 smithi032 bash[21457]: cluster 2024-04-24T17:29:40.677171+0000 mgr.y (mgr.14152) 72 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:42.437 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:42 smithi032 bash[21457]: cluster 2024-04-24T17:29:40.677171+0000 mgr.y (mgr.14152) 72 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:42.437 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:42 smithi032 bash[21457]: cephadm 2024-04-24T17:29:40.734757+0000 mgr.y (mgr.14152) 73 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2024-04-24T17:29:42.437 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:42 smithi032 bash[21457]: cephadm 2024-04-24T17:29:40.734757+0000 mgr.y (mgr.14152) 73 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2024-04-24T17:29:42.437 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:42 smithi032 bash[21457]: cephadm 2024-04-24T17:29:40.738182+0000 mgr.y (mgr.14152) 74 : cephadm [INF] Reconfiguring daemon mon.a on smithi032 2024-04-24T17:29:42.437 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:42 smithi032 bash[21457]: cephadm 2024-04-24T17:29:40.738182+0000 mgr.y (mgr.14152) 74 : cephadm [INF] Reconfiguring daemon mon.a on smithi032 2024-04-24T17:29:42.630 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:42 smithi082 bash[21793]: cluster 2024-04-24T17:29:40.676226+0000 mgr.y (mgr.14152) 71 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:42.630 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:42 smithi082 bash[21793]: cluster 2024-04-24T17:29:40.676226+0000 mgr.y (mgr.14152) 71 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:42.630 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:42 smithi082 bash[21793]: cluster 2024-04-24T17:29:40.677171+0000 mgr.y (mgr.14152) 72 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:42.630 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:42 smithi082 bash[21793]: cluster 2024-04-24T17:29:40.677171+0000 mgr.y (mgr.14152) 72 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:42.630 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:42 smithi082 bash[21793]: cephadm 2024-04-24T17:29:40.734757+0000 mgr.y (mgr.14152) 73 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2024-04-24T17:29:42.630 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:42 smithi082 bash[21793]: cephadm 2024-04-24T17:29:40.734757+0000 mgr.y (mgr.14152) 73 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2024-04-24T17:29:42.631 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:42 smithi082 bash[21793]: cephadm 2024-04-24T17:29:40.738182+0000 mgr.y (mgr.14152) 74 : cephadm [INF] Reconfiguring daemon mon.a on smithi032 2024-04-24T17:29:42.631 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:42 smithi082 bash[21793]: cephadm 2024-04-24T17:29:40.738182+0000 mgr.y (mgr.14152) 74 : cephadm [INF] Reconfiguring daemon mon.a on smithi032 2024-04-24T17:29:43.932 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/mon.b/config 2024-04-24T17:29:44.124 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:43 smithi032 bash[21457]: cluster 2024-04-24T17:29:42.678158+0000 mgr.y (mgr.14152) 75 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:44.124 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:43 smithi032 bash[21457]: cluster 2024-04-24T17:29:42.678158+0000 mgr.y (mgr.14152) 75 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:44.124 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:43 smithi032 bash[21457]: audit 2024-04-24T17:29:42.876198+0000 mon.a (mon.0) 291 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:44.124 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:43 smithi032 bash[21457]: audit 2024-04-24T17:29:42.876198+0000 mon.a (mon.0) 291 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:44.124 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:43 smithi032 bash[21457]: cephadm 2024-04-24T17:29:42.907697+0000 mgr.y (mgr.14152) 76 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-04-24T17:29:44.125 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:43 smithi032 bash[21457]: cephadm 2024-04-24T17:29:42.907697+0000 mgr.y (mgr.14152) 76 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-04-24T17:29:44.125 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:43 smithi032 bash[21457]: audit 2024-04-24T17:29:42.908296+0000 mon.a (mon.0) 292 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T17:29:44.125 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:43 smithi032 bash[21457]: audit 2024-04-24T17:29:42.908296+0000 mon.a (mon.0) 292 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T17:29:44.125 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:43 smithi032 bash[21457]: audit 2024-04-24T17:29:42.909682+0000 mon.a (mon.0) 293 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T17:29:44.125 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:43 smithi032 bash[21457]: audit 2024-04-24T17:29:42.909682+0000 mon.a (mon.0) 293 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T17:29:44.125 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:43 smithi032 bash[21457]: audit 2024-04-24T17:29:42.911059+0000 mon.a (mon.0) 294 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:44.125 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:43 smithi032 bash[21457]: audit 2024-04-24T17:29:42.911059+0000 mon.a (mon.0) 294 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:44.125 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:43 smithi032 bash[21457]: cephadm 2024-04-24T17:29:42.912790+0000 mgr.y (mgr.14152) 77 : cephadm [INF] Reconfiguring daemon mon.c on smithi032 2024-04-24T17:29:44.125 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:43 smithi032 bash[21457]: cephadm 2024-04-24T17:29:42.912790+0000 mgr.y (mgr.14152) 77 : cephadm [INF] Reconfiguring daemon mon.c on smithi032 2024-04-24T17:29:44.125 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:43 smithi032 bash[27083]: cluster 2024-04-24T17:29:42.678158+0000 mgr.y (mgr.14152) 75 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:44.125 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:43 smithi032 bash[27083]: cluster 2024-04-24T17:29:42.678158+0000 mgr.y (mgr.14152) 75 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:44.125 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:43 smithi032 bash[27083]: audit 2024-04-24T17:29:42.876198+0000 mon.a (mon.0) 291 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:44.126 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:43 smithi032 bash[27083]: audit 2024-04-24T17:29:42.876198+0000 mon.a (mon.0) 291 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:44.126 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:43 smithi032 bash[27083]: cephadm 2024-04-24T17:29:42.907697+0000 mgr.y (mgr.14152) 76 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-04-24T17:29:44.126 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:43 smithi032 bash[27083]: cephadm 2024-04-24T17:29:42.907697+0000 mgr.y (mgr.14152) 76 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-04-24T17:29:44.126 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:43 smithi032 bash[27083]: audit 2024-04-24T17:29:42.908296+0000 mon.a (mon.0) 292 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T17:29:44.126 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:43 smithi032 bash[27083]: audit 2024-04-24T17:29:42.908296+0000 mon.a (mon.0) 292 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T17:29:44.126 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:43 smithi032 bash[27083]: audit 2024-04-24T17:29:42.909682+0000 mon.a (mon.0) 293 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T17:29:44.126 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:43 smithi032 bash[27083]: audit 2024-04-24T17:29:42.909682+0000 mon.a (mon.0) 293 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T17:29:44.126 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:43 smithi032 bash[27083]: audit 2024-04-24T17:29:42.911059+0000 mon.a (mon.0) 294 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:44.126 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:43 smithi032 bash[27083]: audit 2024-04-24T17:29:42.911059+0000 mon.a (mon.0) 294 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:44.126 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:43 smithi032 bash[27083]: cephadm 2024-04-24T17:29:42.912790+0000 mgr.y (mgr.14152) 77 : cephadm [INF] Reconfiguring daemon mon.c on smithi032 2024-04-24T17:29:44.126 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:43 smithi032 bash[27083]: cephadm 2024-04-24T17:29:42.912790+0000 mgr.y (mgr.14152) 77 : cephadm [INF] Reconfiguring daemon mon.c on smithi032 2024-04-24T17:29:44.130 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:43 smithi082 bash[21793]: cluster 2024-04-24T17:29:42.678158+0000 mgr.y (mgr.14152) 75 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:44.130 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:43 smithi082 bash[21793]: cluster 2024-04-24T17:29:42.678158+0000 mgr.y (mgr.14152) 75 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:44.130 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:43 smithi082 bash[21793]: audit 2024-04-24T17:29:42.876198+0000 mon.a (mon.0) 291 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:44.130 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:43 smithi082 bash[21793]: audit 2024-04-24T17:29:42.876198+0000 mon.a (mon.0) 291 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:44.130 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:43 smithi082 bash[21793]: cephadm 2024-04-24T17:29:42.907697+0000 mgr.y (mgr.14152) 76 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-04-24T17:29:44.130 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:43 smithi082 bash[21793]: cephadm 2024-04-24T17:29:42.907697+0000 mgr.y (mgr.14152) 76 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-04-24T17:29:44.131 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:43 smithi082 bash[21793]: audit 2024-04-24T17:29:42.908296+0000 mon.a (mon.0) 292 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T17:29:44.131 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:43 smithi082 bash[21793]: audit 2024-04-24T17:29:42.908296+0000 mon.a (mon.0) 292 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T17:29:44.131 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:43 smithi082 bash[21793]: audit 2024-04-24T17:29:42.909682+0000 mon.a (mon.0) 293 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T17:29:44.131 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:43 smithi082 bash[21793]: audit 2024-04-24T17:29:42.909682+0000 mon.a (mon.0) 293 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T17:29:44.131 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:43 smithi082 bash[21793]: audit 2024-04-24T17:29:42.911059+0000 mon.a (mon.0) 294 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:44.131 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:43 smithi082 bash[21793]: audit 2024-04-24T17:29:42.911059+0000 mon.a (mon.0) 294 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:44.131 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:43 smithi082 bash[21793]: cephadm 2024-04-24T17:29:42.912790+0000 mgr.y (mgr.14152) 77 : cephadm [INF] Reconfiguring daemon mon.c on smithi032 2024-04-24T17:29:44.131 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:43 smithi082 bash[21793]: cephadm 2024-04-24T17:29:42.912790+0000 mgr.y (mgr.14152) 77 : cephadm [INF] Reconfiguring daemon mon.c on smithi032 2024-04-24T17:29:44.886 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.881+0000 7f9b86fab700 1 -- 172.21.15.82:0/300323244 >> v1:172.21.15.32:6789/0 conn(0x7f9b801033a0 legacy=0x7f9b801037c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:29:44.886 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.881+0000 7f9b86fab700 1 -- 172.21.15.82:0/300323244 shutdown_connections 2024-04-24T17:29:44.886 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.881+0000 7f9b86fab700 1 -- 172.21.15.82:0/300323244 >> 172.21.15.82:0/300323244 conn(0x7f9b800fcb30 msgr2=0x7f9b800fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:29:44.886 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.881+0000 7f9b86fab700 1 -- 172.21.15.82:0/300323244 shutdown_connections 2024-04-24T17:29:44.886 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.881+0000 7f9b86fab700 1 -- 172.21.15.82:0/300323244 wait complete. 2024-04-24T17:29:44.888 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.881+0000 7f9b86fab700 1 librados: starting msgr at 2024-04-24T17:29:44.888 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.881+0000 7f9b86fab700 1 librados: starting objecter 2024-04-24T17:29:44.888 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.881+0000 7f9b86fab700 1 Processor -- start 2024-04-24T17:29:44.888 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.881+0000 7f9b86fab700 1 -- start start 2024-04-24T17:29:44.888 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.881+0000 7f9b86fab700 1 librados: setting wanted keys 2024-04-24T17:29:44.888 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.881+0000 7f9b86fab700 1 librados: calling monclient init 2024-04-24T17:29:44.888 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.881+0000 7f9b86fab700 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9b801a32c0 con 0x7f9b8019f650 2024-04-24T17:29:44.889 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.881+0000 7f9b86fab700 1 -- --> v1:172.21.15.82:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9b801a44e0 con 0x7f9b801033a0 2024-04-24T17:29:44.889 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.881+0000 7f9b86fab700 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9b801a5720 con 0x7f9b80198160 2024-04-24T17:29:44.889 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.881+0000 7f9b84d47700 1 --1- >> v1:172.21.15.82:6789/0 conn(0x7f9b801033a0 0x7f9b80197a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.82:6789/0 says I am v1:172.21.15.82:48284/0 (socket says 172.21.15.82:48284) 2024-04-24T17:29:44.889 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.881+0000 7f9b84d47700 1 -- 172.21.15.82:0/3951235483 learned_addr learned my addr 172.21.15.82:0/3951235483 (peer_addr_for_me v1:172.21.15.82:0/0) 2024-04-24T17:29:44.889 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.881+0000 7f9b7dffb700 10 client.?.objecter ms_handle_connect 0x7f9b801033a0 2024-04-24T17:29:44.889 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.881+0000 7f9b7dffb700 10 client.?.objecter resend_mon_ops 2024-04-24T17:29:44.889 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.885+0000 7f9b7dffb700 1 -- 172.21.15.82:0/3951235483 <== mon.1 v1:172.21.15.82:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3016399371 0 0) 0x7f9b801a44e0 con 0x7f9b801033a0 2024-04-24T17:29:44.889 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.885+0000 7f9b7dffb700 1 -- 172.21.15.82:0/3951235483 --> v1:172.21.15.82:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b70003c40 con 0x7f9b801033a0 2024-04-24T17:29:44.889 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.885+0000 7f9b7dffb700 10 client.?.objecter ms_handle_connect 0x7f9b80198160 2024-04-24T17:29:44.889 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.885+0000 7f9b7dffb700 10 client.?.objecter resend_mon_ops 2024-04-24T17:29:44.889 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.885+0000 7f9b7dffb700 10 client.?.objecter ms_handle_connect 0x7f9b8019f650 2024-04-24T17:29:44.889 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.885+0000 7f9b7dffb700 10 client.?.objecter resend_mon_ops 2024-04-24T17:29:44.889 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.885+0000 7f9b7dffb700 1 -- 172.21.15.82:0/3951235483 <== mon.1 v1:172.21.15.82:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 434296399 0 0) 0x7f9b70003c40 con 0x7f9b801033a0 2024-04-24T17:29:44.890 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.885+0000 7f9b7dffb700 1 -- 172.21.15.82:0/3951235483 --> v1:172.21.15.82:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b801a44e0 con 0x7f9b801033a0 2024-04-24T17:29:44.890 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.885+0000 7f9b7dffb700 1 -- 172.21.15.82:0/3951235483 <== mon.1 v1:172.21.15.82:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 512931633 0 0) 0x7f9b74004490 con 0x7f9b801033a0 2024-04-24T17:29:44.890 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.885+0000 7f9b7dffb700 1 -- 172.21.15.82:0/3951235483 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 764885628 0 0) 0x7f9b801a5720 con 0x7f9b80198160 2024-04-24T17:29:44.890 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.885+0000 7f9b7dffb700 1 -- 172.21.15.82:0/3951235483 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b70003c40 con 0x7f9b80198160 2024-04-24T17:29:44.890 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.885+0000 7f9b7dffb700 1 -- 172.21.15.82:0/3951235483 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2374972243 0 0) 0x7f9b801a32c0 con 0x7f9b8019f650 2024-04-24T17:29:44.890 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.885+0000 7f9b7dffb700 1 -- 172.21.15.82:0/3951235483 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b801a5720 con 0x7f9b8019f650 2024-04-24T17:29:44.890 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.885+0000 7f9b7dffb700 1 -- 172.21.15.82:0/3951235483 <== mon.1 v1:172.21.15.82:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 741524077 0 0) 0x7f9b801a44e0 con 0x7f9b801033a0 2024-04-24T17:29:44.890 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.885+0000 7f9b7dffb700 1 -- 172.21.15.82:0/3951235483 >> v1:172.21.15.32:6790/0 conn(0x7f9b80198160 legacy=0x7f9b8019df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:29:44.890 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.885+0000 7f9b7dffb700 1 -- 172.21.15.82:0/3951235483 >> v1:172.21.15.32:6789/0 conn(0x7f9b8019f650 legacy=0x7f9b801a1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:29:44.890 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.885+0000 7f9b7dffb700 1 -- 172.21.15.82:0/3951235483 --> v1:172.21.15.82:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b801a6960 con 0x7f9b801033a0 2024-04-24T17:29:44.890 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.885+0000 7f9b86fab700 1 -- 172.21.15.82:0/3951235483 --> v1:172.21.15.82:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9b801a5950 con 0x7f9b801033a0 2024-04-24T17:29:44.890 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.885+0000 7f9b86fab700 10 client.24103.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-24T17:29:44.890 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.885+0000 7f9b86fab700 1 -- 172.21.15.82:0/3951235483 --> v1:172.21.15.82:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f9b801a5e90 con 0x7f9b801033a0 2024-04-24T17:29:44.890 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.885+0000 7f9b86fab700 1 librados: init done 2024-04-24T17:29:44.890 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.885+0000 7f9b7dffb700 1 -- 172.21.15.82:0/3951235483 <== mon.1 v1:172.21.15.82:6789/0 5 ==== config(25 keys) ==== 1042+0+0 (unknown 2768897130 0 0) 0x7f9b74003f40 con 0x7f9b801033a0 2024-04-24T17:29:44.891 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.885+0000 7f9b7dffb700 1 -- 172.21.15.82:0/3951235483 <== mon.1 v1:172.21.15.82:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 512931633 0 0) 0x7f9b74004df0 con 0x7f9b801033a0 2024-04-24T17:29:44.891 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.885+0000 7f9b7dffb700 1 -- 172.21.15.82:0/3951235483 <== mon.1 v1:172.21.15.82:6789/0 7 ==== mgrmap(e 13) ==== 46579+0+0 (unknown 583022566 0 0) 0x7f9b74005030 con 0x7f9b801033a0 2024-04-24T17:29:44.891 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.885+0000 7f9b86fab700 1 -- 172.21.15.82:0/3951235483 --> v1:172.21.15.82:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b64005150 con 0x7f9b801033a0 2024-04-24T17:29:44.891 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.885+0000 7f9b7dffb700 1 -- 172.21.15.82:0/3951235483 <== mon.1 v1:172.21.15.82:6789/0 8 ==== osd_map(4..4 src has 1..4) ==== 1069+0+0 (unknown 1280957937 0 0) 0x7f9b74048020 con 0x7f9b801033a0 2024-04-24T17:29:44.892 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.885+0000 7f9b7dffb700 10 client.24103.objecter ms_dispatch 0x7f9b80001550 osd_map(4..4 src has 1..4) 2024-04-24T17:29:44.892 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.885+0000 7f9b7dffb700 3 client.24103.objecter handle_osd_map got epochs [4,4] > 0 2024-04-24T17:29:44.892 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.885+0000 7f9b7dffb700 3 client.24103.objecter handle_osd_map decoding full epoch 4 2024-04-24T17:29:44.892 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.885+0000 7f9b7dffb700 20 client.24103.objecter dump_active .. 0 homeless 2024-04-24T17:29:44.892 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.885+0000 7f9b7dffb700 10 client.24103.objecter ms_handle_connect 0x7f9b7003c530 2024-04-24T17:29:44.894 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:44.889+0000 7f9b7dffb700 1 -- 172.21.15.82:0/3951235483 <== mon.1 v1:172.21.15.82:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188728 (unknown 1092875540 0 2656817166) 0x7f9b74017700 con 0x7f9b801033a0 2024-04-24T17:29:45.113 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:45.109+0000 7f9b86fab700 1 -- 172.21.15.82:0/3951235483 --> v1:172.21.15.32:6800/1847389373 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi032=y;smithi082=x", "target": ["mon-mgr", ""]}) -- 0x7f9b64000bc0 con 0x7f9b7003c530 2024-04-24T17:29:45.127 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:45.121+0000 7f9b7dffb700 1 -- 172.21.15.82:0/3951235483 <== mgr.14152 v1:172.21.15.32:6800/1847389373 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (unknown 0 0 325935098) 0x7f9b64000bc0 con 0x7f9b7003c530 2024-04-24T17:29:45.127 INFO:teuthology.orchestra.run.smithi082.stdout:Scheduled mgr update... 2024-04-24T17:29:45.130 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:45.125+0000 7f9b86fab700 10 librados: watch_flush enter 2024-04-24T17:29:45.130 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:45.125+0000 7f9b86fab700 10 librados: watch_flush exit 2024-04-24T17:29:45.130 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:45.125+0000 7f9b86fab700 20 client.24103.objecter shutdown clearing up homeless session... 2024-04-24T17:29:45.130 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:45.125+0000 7f9b86fab700 10 client.24103.objecter successfully canceled tick 2024-04-24T17:29:45.130 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:45.125+0000 7f9b86fab700 1 -- 172.21.15.82:0/3951235483 >> v1:172.21.15.32:6800/1847389373 conn(0x7f9b7003c530 legacy=0x7f9b7003e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:29:45.130 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:45.125+0000 7f9b86fab700 1 -- 172.21.15.82:0/3951235483 >> v1:172.21.15.82:6789/0 conn(0x7f9b801033a0 legacy=0x7f9b80197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T17:29:45.130 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:45.125+0000 7f9b86fab700 1 -- 172.21.15.82:0/3951235483 shutdown_connections 2024-04-24T17:29:45.130 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:45.125+0000 7f9b86fab700 1 -- 172.21.15.82:0/3951235483 >> 172.21.15.82:0/3951235483 conn(0x7f9b800fcb30 msgr2=0x7f9b80104930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T17:29:45.130 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:45.125+0000 7f9b86fab700 1 -- 172.21.15.82:0/3951235483 shutdown_connections 2024-04-24T17:29:45.130 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:45.125+0000 7f9b86fab700 1 -- 172.21.15.82:0/3951235483 wait complete. 2024-04-24T17:29:45.130 INFO:teuthology.orchestra.run.smithi082.stderr:2024-04-24T17:29:45.125+0000 7f9b86fab700 1 librados: shutdown 2024-04-24T17:29:45.917 DEBUG:teuthology.orchestra.run.smithi082:mgr.x> sudo journalctl -f -n 0 -u ceph-a7981b84-025f-11ef-bc93-c7b262605968@mgr.x.service 2024-04-24T17:29:45.919 INFO:tasks.cephadm:Deploying OSDs... 2024-04-24T17:29:45.919 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-24T17:29:45.919 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/scratch_devs of=/dev/stdout 2024-04-24T17:29:45.926 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-24T17:29:45.926 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_1 2024-04-24T17:29:45.975 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-24T17:29:45.975 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T17:29:45.975 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 872 Links: 1 2024-04-24T17:29:45.975 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T17:29:45.975 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-04-24 17:20:10.737869814 +0000 2024-04-24T17:29:45.975 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-04-24 17:20:10.477872136 +0000 2024-04-24T17:29:45.976 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-04-24 17:20:10.477872136 +0000 2024-04-24T17:29:45.976 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: - 2024-04-24T17:29:45.976 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-24T17:29:46.029 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-04-24T17:29:46.029 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-04-24T17:29:46.029 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000269629 s, 1.9 MB/s 2024-04-24T17:29:46.031 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-24T17:29:46.079 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_2 2024-04-24T17:29:46.126 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-24T17:29:46.126 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T17:29:46.126 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 891 Links: 1 2024-04-24T17:29:46.126 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T17:29:46.126 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-04-24 17:20:11.209865598 +0000 2024-04-24T17:29:46.126 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-04-24 17:20:10.953867885 +0000 2024-04-24T17:29:46.126 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-04-24 17:20:10.953867885 +0000 2024-04-24T17:29:46.126 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: - 2024-04-24T17:29:46.127 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-24T17:29:46.169 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:45 smithi082 bash[21793]: cluster 2024-04-24T17:29:44.679129+0000 mgr.y (mgr.14152) 78 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:46.169 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:45 smithi082 bash[21793]: cluster 2024-04-24T17:29:44.679129+0000 mgr.y (mgr.14152) 78 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:46.169 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:45 smithi082 bash[21793]: audit 2024-04-24T17:29:44.932274+0000 mon.a (mon.0) 295 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:46.169 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:45 smithi082 bash[21793]: audit 2024-04-24T17:29:44.932274+0000 mon.a (mon.0) 295 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:46.169 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:45 smithi082 bash[21793]: cephadm 2024-04-24T17:29:44.934012+0000 mgr.y (mgr.14152) 79 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-04-24T17:29:46.169 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:45 smithi082 bash[21793]: cephadm 2024-04-24T17:29:44.934012+0000 mgr.y (mgr.14152) 79 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-04-24T17:29:46.169 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:45 smithi082 bash[21793]: audit 2024-04-24T17:29:44.934597+0000 mon.a (mon.0) 296 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T17:29:46.169 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:45 smithi082 bash[21793]: audit 2024-04-24T17:29:44.934597+0000 mon.a (mon.0) 296 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T17:29:46.169 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:45 smithi082 bash[21793]: audit 2024-04-24T17:29:44.936057+0000 mon.a (mon.0) 297 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T17:29:46.170 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:45 smithi082 bash[21793]: audit 2024-04-24T17:29:44.936057+0000 mon.a (mon.0) 297 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T17:29:46.170 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:45 smithi082 bash[21793]: audit 2024-04-24T17:29:44.937353+0000 mon.a (mon.0) 298 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:46.170 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:45 smithi082 bash[21793]: audit 2024-04-24T17:29:44.937353+0000 mon.a (mon.0) 298 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:46.170 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:45 smithi082 bash[21793]: cephadm 2024-04-24T17:29:44.938894+0000 mgr.y (mgr.14152) 80 : cephadm [INF] Reconfiguring daemon mon.b on smithi082 2024-04-24T17:29:46.170 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:45 smithi082 bash[21793]: cephadm 2024-04-24T17:29:44.938894+0000 mgr.y (mgr.14152) 80 : cephadm [INF] Reconfiguring daemon mon.b on smithi082 2024-04-24T17:29:46.170 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:45 smithi082 bash[21793]: audit 2024-04-24T17:29:45.116937+0000 mgr.y (mgr.14152) 81 : audit [DBG] from='client.24103 v1:172.21.15.82:0/3951235483' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi032=y;smithi082=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T17:29:46.170 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:45 smithi082 bash[21793]: audit 2024-04-24T17:29:45.116937+0000 mgr.y (mgr.14152) 81 : audit [DBG] from='client.24103 v1:172.21.15.82:0/3951235483' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi032=y;smithi082=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T17:29:46.170 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:45 smithi082 bash[21793]: cephadm 2024-04-24T17:29:45.120472+0000 mgr.y (mgr.14152) 82 : cephadm [INF] Saving service mgr spec with placement smithi032=y;smithi082=x;count:2 2024-04-24T17:29:46.170 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:45 smithi082 bash[21793]: cephadm 2024-04-24T17:29:45.120472+0000 mgr.y (mgr.14152) 82 : cephadm [INF] Saving service mgr spec with placement smithi032=y;smithi082=x;count:2 2024-04-24T17:29:46.170 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:45 smithi082 bash[21793]: audit 2024-04-24T17:29:45.127379+0000 mon.a (mon.0) 299 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:46.170 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:45 smithi082 bash[21793]: audit 2024-04-24T17:29:45.127379+0000 mon.a (mon.0) 299 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:46.181 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:45 smithi032 bash[27083]: cluster 2024-04-24T17:29:44.679129+0000 mgr.y (mgr.14152) 78 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:46.181 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:45 smithi032 bash[27083]: cluster 2024-04-24T17:29:44.679129+0000 mgr.y (mgr.14152) 78 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:46.181 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:45 smithi032 bash[27083]: audit 2024-04-24T17:29:44.932274+0000 mon.a (mon.0) 295 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:46.181 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:45 smithi032 bash[27083]: audit 2024-04-24T17:29:44.932274+0000 mon.a (mon.0) 295 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:46.181 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:45 smithi032 bash[27083]: cephadm 2024-04-24T17:29:44.934012+0000 mgr.y (mgr.14152) 79 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-04-24T17:29:46.181 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:45 smithi032 bash[27083]: cephadm 2024-04-24T17:29:44.934012+0000 mgr.y (mgr.14152) 79 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-04-24T17:29:46.181 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:45 smithi032 bash[27083]: audit 2024-04-24T17:29:44.934597+0000 mon.a (mon.0) 296 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T17:29:46.181 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:45 smithi032 bash[27083]: audit 2024-04-24T17:29:44.934597+0000 mon.a (mon.0) 296 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T17:29:46.181 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:45 smithi032 bash[27083]: audit 2024-04-24T17:29:44.936057+0000 mon.a (mon.0) 297 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T17:29:46.182 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:45 smithi032 bash[27083]: audit 2024-04-24T17:29:44.936057+0000 mon.a (mon.0) 297 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T17:29:46.182 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:45 smithi032 bash[27083]: audit 2024-04-24T17:29:44.937353+0000 mon.a (mon.0) 298 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:46.182 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:45 smithi032 bash[27083]: audit 2024-04-24T17:29:44.937353+0000 mon.a (mon.0) 298 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:46.182 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:45 smithi032 bash[27083]: cephadm 2024-04-24T17:29:44.938894+0000 mgr.y (mgr.14152) 80 : cephadm [INF] Reconfiguring daemon mon.b on smithi082 2024-04-24T17:29:46.182 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:45 smithi032 bash[27083]: cephadm 2024-04-24T17:29:44.938894+0000 mgr.y (mgr.14152) 80 : cephadm [INF] Reconfiguring daemon mon.b on smithi082 2024-04-24T17:29:46.182 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:45 smithi032 bash[27083]: audit 2024-04-24T17:29:45.116937+0000 mgr.y (mgr.14152) 81 : audit [DBG] from='client.24103 v1:172.21.15.82:0/3951235483' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi032=y;smithi082=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T17:29:46.182 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:45 smithi032 bash[27083]: audit 2024-04-24T17:29:45.116937+0000 mgr.y (mgr.14152) 81 : audit [DBG] from='client.24103 v1:172.21.15.82:0/3951235483' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi032=y;smithi082=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T17:29:46.182 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:45 smithi032 bash[27083]: cephadm 2024-04-24T17:29:45.120472+0000 mgr.y (mgr.14152) 82 : cephadm [INF] Saving service mgr spec with placement smithi032=y;smithi082=x;count:2 2024-04-24T17:29:46.182 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:45 smithi032 bash[27083]: cephadm 2024-04-24T17:29:45.120472+0000 mgr.y (mgr.14152) 82 : cephadm [INF] Saving service mgr spec with placement smithi032=y;smithi082=x;count:2 2024-04-24T17:29:46.182 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:45 smithi032 bash[27083]: audit 2024-04-24T17:29:45.127379+0000 mon.a (mon.0) 299 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:46.182 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:45 smithi032 bash[27083]: audit 2024-04-24T17:29:45.127379+0000 mon.a (mon.0) 299 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:46.183 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:45 smithi032 bash[21457]: cluster 2024-04-24T17:29:44.679129+0000 mgr.y (mgr.14152) 78 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:46.183 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:45 smithi032 bash[21457]: cluster 2024-04-24T17:29:44.679129+0000 mgr.y (mgr.14152) 78 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:46.183 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:45 smithi032 bash[21457]: audit 2024-04-24T17:29:44.932274+0000 mon.a (mon.0) 295 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:46.183 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:45 smithi032 bash[21457]: audit 2024-04-24T17:29:44.932274+0000 mon.a (mon.0) 295 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:46.183 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:45 smithi032 bash[21457]: cephadm 2024-04-24T17:29:44.934012+0000 mgr.y (mgr.14152) 79 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-04-24T17:29:46.183 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:45 smithi032 bash[21457]: cephadm 2024-04-24T17:29:44.934012+0000 mgr.y (mgr.14152) 79 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-04-24T17:29:46.183 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:45 smithi032 bash[21457]: audit 2024-04-24T17:29:44.934597+0000 mon.a (mon.0) 296 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T17:29:46.183 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:45 smithi032 bash[21457]: audit 2024-04-24T17:29:44.934597+0000 mon.a (mon.0) 296 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T17:29:46.183 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:45 smithi032 bash[21457]: audit 2024-04-24T17:29:44.936057+0000 mon.a (mon.0) 297 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T17:29:46.183 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:45 smithi032 bash[21457]: audit 2024-04-24T17:29:44.936057+0000 mon.a (mon.0) 297 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T17:29:46.183 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:45 smithi032 bash[21457]: audit 2024-04-24T17:29:44.937353+0000 mon.a (mon.0) 298 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:46.183 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:45 smithi032 bash[21457]: audit 2024-04-24T17:29:44.937353+0000 mon.a (mon.0) 298 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:46.183 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:45 smithi032 bash[21457]: cephadm 2024-04-24T17:29:44.938894+0000 mgr.y (mgr.14152) 80 : cephadm [INF] Reconfiguring daemon mon.b on smithi082 2024-04-24T17:29:46.184 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:45 smithi032 bash[21457]: cephadm 2024-04-24T17:29:44.938894+0000 mgr.y (mgr.14152) 80 : cephadm [INF] Reconfiguring daemon mon.b on smithi082 2024-04-24T17:29:46.184 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:45 smithi032 bash[21457]: audit 2024-04-24T17:29:45.116937+0000 mgr.y (mgr.14152) 81 : audit [DBG] from='client.24103 v1:172.21.15.82:0/3951235483' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi032=y;smithi082=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T17:29:46.184 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:45 smithi032 bash[21457]: audit 2024-04-24T17:29:45.116937+0000 mgr.y (mgr.14152) 81 : audit [DBG] from='client.24103 v1:172.21.15.82:0/3951235483' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi032=y;smithi082=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T17:29:46.184 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:45 smithi032 bash[21457]: cephadm 2024-04-24T17:29:45.120472+0000 mgr.y (mgr.14152) 82 : cephadm [INF] Saving service mgr spec with placement smithi032=y;smithi082=x;count:2 2024-04-24T17:29:46.184 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:45 smithi032 bash[21457]: cephadm 2024-04-24T17:29:45.120472+0000 mgr.y (mgr.14152) 82 : cephadm [INF] Saving service mgr spec with placement smithi032=y;smithi082=x;count:2 2024-04-24T17:29:46.184 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:45 smithi032 bash[21457]: audit 2024-04-24T17:29:45.127379+0000 mon.a (mon.0) 299 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:46.184 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:45 smithi032 bash[21457]: audit 2024-04-24T17:29:45.127379+0000 mon.a (mon.0) 299 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:46.185 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-04-24T17:29:46.185 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-04-24T17:29:46.185 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.00020712 s, 2.5 MB/s 2024-04-24T17:29:46.185 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-24T17:29:46.236 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_3 2024-04-24T17:29:46.282 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-24T17:29:46.282 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T17:29:46.282 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 906 Links: 1 2024-04-24T17:29:46.282 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T17:29:46.282 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-04-24 17:20:11.713861096 +0000 2024-04-24T17:29:46.282 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-04-24 17:20:11.453863419 +0000 2024-04-24T17:29:46.282 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-04-24 17:20:11.453863419 +0000 2024-04-24T17:29:46.282 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: - 2024-04-24T17:29:46.283 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-24T17:29:46.337 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-04-24T17:29:46.337 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-04-24T17:29:46.337 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000277023 s, 1.8 MB/s 2024-04-24T17:29:46.338 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-24T17:29:46.391 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_4 2024-04-24T17:29:46.438 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-24T17:29:46.438 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T17:29:46.439 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 923 Links: 1 2024-04-24T17:29:46.439 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T17:29:46.439 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-04-24 17:20:12.197856773 +0000 2024-04-24T17:29:46.439 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-04-24 17:20:11.937859095 +0000 2024-04-24T17:29:46.439 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-04-24 17:20:11.937859095 +0000 2024-04-24T17:29:46.439 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: - 2024-04-24T17:29:46.439 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-24T17:29:46.493 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-04-24T17:29:46.493 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-04-24T17:29:46.493 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000269646 s, 1.9 MB/s 2024-04-24T17:29:46.494 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-24T17:29:46.544 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-04-24T17:29:46.544 DEBUG:teuthology.orchestra.run.smithi082:> dd if=/scratch_devs of=/dev/stdout 2024-04-24T17:29:46.550 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-24T17:29:46.550 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_1 2024-04-24T17:29:46.596 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-24T17:29:46.596 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T17:29:46.596 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 5h/5d Inode: 880 Links: 1 2024-04-24T17:29:46.596 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T17:29:46.596 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-04-24 17:20:06.564849989 +0000 2024-04-24T17:29:46.596 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-04-24 17:20:06.292852419 +0000 2024-04-24T17:29:46.596 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-04-24 17:20:06.292852419 +0000 2024-04-24T17:29:46.596 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: - 2024-04-24T17:29:46.597 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-24T17:29:46.651 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-04-24T17:29:46.651 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-04-24T17:29:46.651 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000410693 s, 1.2 MB/s 2024-04-24T17:29:46.652 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-24T17:29:46.701 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_2 2024-04-24T17:29:46.748 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-24T17:29:46.748 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T17:29:46.748 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 5h/5d Inode: 898 Links: 1 2024-04-24T17:29:46.748 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T17:29:46.748 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-04-24 17:20:06.800847881 +0000 2024-04-24T17:29:46.748 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-04-24 17:20:06.796847917 +0000 2024-04-24T17:29:46.748 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-04-24 17:20:06.796847917 +0000 2024-04-24T17:29:46.748 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: - 2024-04-24T17:29:46.749 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-24T17:29:46.803 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-04-24T17:29:46.803 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-04-24T17:29:46.803 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.00042334 s, 1.2 MB/s 2024-04-24T17:29:46.804 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-24T17:29:46.853 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_3 2024-04-24T17:29:46.900 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-24T17:29:46.900 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T17:29:46.900 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 5h/5d Inode: 912 Links: 1 2024-04-24T17:29:46.900 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T17:29:46.900 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-04-24 17:20:07.560841093 +0000 2024-04-24T17:29:46.900 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-04-24 17:20:07.288843522 +0000 2024-04-24T17:29:46.900 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-04-24 17:20:07.288843522 +0000 2024-04-24T17:29:46.900 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: - 2024-04-24T17:29:46.900 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-24T17:29:46.955 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-04-24T17:29:46.955 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-04-24T17:29:46.955 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000436813 s, 1.2 MB/s 2024-04-24T17:29:46.956 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-24T17:29:47.005 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_4 2024-04-24T17:29:47.052 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-24T17:29:47.052 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T17:29:47.052 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 5h/5d Inode: 929 Links: 1 2024-04-24T17:29:47.052 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T17:29:47.052 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-04-24 17:20:08.036836841 +0000 2024-04-24T17:29:47.052 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-04-24 17:20:07.792839020 +0000 2024-04-24T17:29:47.052 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-04-24 17:20:07.792839020 +0000 2024-04-24T17:29:47.052 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: - 2024-04-24T17:29:47.052 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-24T17:29:47.106 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-04-24T17:29:47.106 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-04-24T17:29:47.106 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000305164 s, 1.7 MB/s 2024-04-24T17:29:47.107 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-24T17:29:47.157 INFO:tasks.cephadm:Deploying osd.0 on smithi032 with /dev/vg_nvme/lv_4... 2024-04-24T17:29:47.157 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a7981b84-025f-11ef-bc93-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-04-24T17:29:48.252 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:47 smithi032 bash[27083]: cluster 2024-04-24T17:29:46.680075+0000 mgr.y (mgr.14152) 83 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:48.252 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:47 smithi032 bash[27083]: cluster 2024-04-24T17:29:46.680075+0000 mgr.y (mgr.14152) 83 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:48.252 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:47 smithi032 bash[27083]: audit 2024-04-24T17:29:46.964092+0000 mon.a (mon.0) 300 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:48.252 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:47 smithi032 bash[27083]: audit 2024-04-24T17:29:46.964092+0000 mon.a (mon.0) 300 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:48.253 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:47 smithi032 bash[27083]: audit 2024-04-24T17:29:47.011043+0000 mon.a (mon.0) 301 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T17:29:48.253 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:47 smithi032 bash[27083]: audit 2024-04-24T17:29:47.011043+0000 mon.a (mon.0) 301 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T17:29:48.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:47 smithi032 bash[21457]: cluster 2024-04-24T17:29:46.680075+0000 mgr.y (mgr.14152) 83 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:48.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:47 smithi032 bash[21457]: cluster 2024-04-24T17:29:46.680075+0000 mgr.y (mgr.14152) 83 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:48.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:47 smithi032 bash[21457]: audit 2024-04-24T17:29:46.964092+0000 mon.a (mon.0) 300 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:48.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:47 smithi032 bash[21457]: audit 2024-04-24T17:29:46.964092+0000 mon.a (mon.0) 300 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:48.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:47 smithi032 bash[21457]: audit 2024-04-24T17:29:47.011043+0000 mon.a (mon.0) 301 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T17:29:48.253 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:47 smithi032 bash[21457]: audit 2024-04-24T17:29:47.011043+0000 mon.a (mon.0) 301 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T17:29:48.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:47 smithi082 bash[21793]: cluster 2024-04-24T17:29:46.680075+0000 mgr.y (mgr.14152) 83 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:48.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:47 smithi082 bash[21793]: cluster 2024-04-24T17:29:46.680075+0000 mgr.y (mgr.14152) 83 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:48.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:47 smithi082 bash[21793]: audit 2024-04-24T17:29:46.964092+0000 mon.a (mon.0) 300 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:48.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:47 smithi082 bash[21793]: audit 2024-04-24T17:29:46.964092+0000 mon.a (mon.0) 300 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:48.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:47 smithi082 bash[21793]: audit 2024-04-24T17:29:47.011043+0000 mon.a (mon.0) 301 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T17:29:48.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:47 smithi082 bash[21793]: audit 2024-04-24T17:29:47.011043+0000 mon.a (mon.0) 301 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T17:29:50.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:49 smithi032 bash[21457]: cluster 2024-04-24T17:29:48.681072+0000 mgr.y (mgr.14152) 84 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:50.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:49 smithi032 bash[21457]: cluster 2024-04-24T17:29:48.681072+0000 mgr.y (mgr.14152) 84 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:50.252 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:49 smithi032 bash[27083]: cluster 2024-04-24T17:29:48.681072+0000 mgr.y (mgr.14152) 84 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:50.252 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:49 smithi032 bash[27083]: cluster 2024-04-24T17:29:48.681072+0000 mgr.y (mgr.14152) 84 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:50.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:49 smithi082 bash[21793]: cluster 2024-04-24T17:29:48.681072+0000 mgr.y (mgr.14152) 84 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:50.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:49 smithi082 bash[21793]: cluster 2024-04-24T17:29:48.681072+0000 mgr.y (mgr.14152) 84 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:51.886 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T17:29:52.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:52 smithi032 bash[21457]: cluster 2024-04-24T17:29:50.682046+0000 mgr.y (mgr.14152) 85 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:52.252 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:52 smithi032 bash[21457]: cluster 2024-04-24T17:29:50.682046+0000 mgr.y (mgr.14152) 85 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:52.252 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:52 smithi032 bash[27083]: cluster 2024-04-24T17:29:50.682046+0000 mgr.y (mgr.14152) 85 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:52.252 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:52 smithi032 bash[27083]: cluster 2024-04-24T17:29:50.682046+0000 mgr.y (mgr.14152) 85 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:52.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:52 smithi082 bash[21793]: cluster 2024-04-24T17:29:50.682046+0000 mgr.y (mgr.14152) 85 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:52.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:52 smithi082 bash[21793]: cluster 2024-04-24T17:29:50.682046+0000 mgr.y (mgr.14152) 85 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:54.358 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:54 smithi032 bash[21457]: cluster 2024-04-24T17:29:52.683030+0000 mgr.y (mgr.14152) 86 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:54.358 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:54 smithi032 bash[21457]: cluster 2024-04-24T17:29:52.683030+0000 mgr.y (mgr.14152) 86 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:54.358 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:54 smithi032 bash[27083]: cluster 2024-04-24T17:29:52.683030+0000 mgr.y (mgr.14152) 86 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:54.358 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:54 smithi032 bash[27083]: cluster 2024-04-24T17:29:52.683030+0000 mgr.y (mgr.14152) 86 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:54.630 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:54 smithi082 bash[21793]: cluster 2024-04-24T17:29:52.683030+0000 mgr.y (mgr.14152) 86 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:54.630 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:54 smithi082 bash[21793]: cluster 2024-04-24T17:29:52.683030+0000 mgr.y (mgr.14152) 86 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T17:29:55.055 INFO:teuthology.orchestra.run.smithi032.stderr:Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc -e NODE_NAME=smithi032 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpbl8fnlnv:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp8fhjs1ap:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc lvm zap /dev/vg_nvme/lv_4 2024-04-24T17:29:55.055 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/docker: stderr docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:55.055 INFO:teuthology.orchestra.run.smithi032.stderr:/usr/bin/docker: stderr See 'docker run --help'. 2024-04-24T17:29:55.055 INFO:teuthology.orchestra.run.smithi032.stderr:Traceback (most recent call last): 2024-04-24T17:29:55.055 INFO:teuthology.orchestra.run.smithi032.stderr: File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main 2024-04-24T17:29:55.056 INFO:teuthology.orchestra.run.smithi032.stderr: return _run_code(code, main_globals, None, 2024-04-24T17:29:55.056 INFO:teuthology.orchestra.run.smithi032.stderr: File "/usr/lib/python3.10/runpy.py", line 86, in _run_code 2024-04-24T17:29:55.056 INFO:teuthology.orchestra.run.smithi032.stderr: exec(code, run_globals) 2024-04-24T17:29:55.056 INFO:teuthology.orchestra.run.smithi032.stderr: File "/tmp/tmp7kwbd9jv.cephadm.build/app/__main__.py", line 5642, in 2024-04-24T17:29:55.056 INFO:teuthology.orchestra.run.smithi032.stderr: File "/tmp/tmp7kwbd9jv.cephadm.build/app/__main__.py", line 5630, in main 2024-04-24T17:29:55.056 INFO:teuthology.orchestra.run.smithi032.stderr: File "/tmp/tmp7kwbd9jv.cephadm.build/app/__main__.py", line 408, in _infer_config 2024-04-24T17:29:55.056 INFO:teuthology.orchestra.run.smithi032.stderr: File "/tmp/tmp7kwbd9jv.cephadm.build/app/__main__.py", line 323, in _infer_fsid 2024-04-24T17:29:55.056 INFO:teuthology.orchestra.run.smithi032.stderr: File "/tmp/tmp7kwbd9jv.cephadm.build/app/__main__.py", line 436, in _infer_image 2024-04-24T17:29:55.056 INFO:teuthology.orchestra.run.smithi032.stderr: File "/tmp/tmp7kwbd9jv.cephadm.build/app/__main__.py", line 310, in _validate_fsid 2024-04-24T17:29:55.056 INFO:teuthology.orchestra.run.smithi032.stderr: File "/tmp/tmp7kwbd9jv.cephadm.build/app/__main__.py", line 3380, in command_ceph_volume 2024-04-24T17:29:55.056 INFO:teuthology.orchestra.run.smithi032.stderr: File "/tmp/tmp7kwbd9jv.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-24T17:29:55.056 INFO:teuthology.orchestra.run.smithi032.stderr:RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc -e NODE_NAME=smithi032 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/run/ceph:z -v /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:slave -v /tmp/ceph-tmpbl8fnlnv:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp8fhjs1ap:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc lvm zap /dev/vg_nvme/lv_4: docker: Error response from daemon: invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: "/var/lib/docker", bind mount source: "/", propagation: "slave". 2024-04-24T17:29:55.056 INFO:teuthology.orchestra.run.smithi032.stderr:See 'docker run --help'. 2024-04-24T17:29:55.056 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-04-24T17:29:55.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T17:29:55.088 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_5904d29475f5be602879d9fb26280e89b808d5cc/qa/tasks/cephadm.py", line 1091, in ceph_osds remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi032 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a7981b84-025f-11ef-bc93-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4' 2024-04-24T17:29:55.090 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-24T17:29:55.103 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-24T17:29:55.116 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-24T17:29:55.116 DEBUG:teuthology.orchestra.run.smithi032:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-24T17:29:55.150 DEBUG:teuthology.orchestra.run.smithi082:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-24T17:29:55.163 INFO:tasks.cephadm:Stopping all daemons... 2024-04-24T17:29:55.164 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-24T17:29:55.164 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-a7981b84-025f-11ef-bc93-c7b262605968@mon.a 2024-04-24T17:29:55.366 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:55 smithi032 bash[27083]: audit 2024-04-24T17:29:54.805546+0000 mon.a (mon.0) 302 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:55.366 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:55 smithi032 bash[27083]: audit 2024-04-24T17:29:54.805546+0000 mon.a (mon.0) 302 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:55.366 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:55 smithi032 bash[27083]: audit 2024-04-24T17:29:54.806706+0000 mon.a (mon.0) 303 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T17:29:55.366 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:55 smithi032 bash[27083]: audit 2024-04-24T17:29:54.806706+0000 mon.a (mon.0) 303 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T17:29:55.366 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:55 smithi032 bash[27083]: audit 2024-04-24T17:29:54.812397+0000 mon.a (mon.0) 304 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:55.366 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:55 smithi032 bash[27083]: audit 2024-04-24T17:29:54.812397+0000 mon.a (mon.0) 304 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:55.366 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:55 smithi032 bash[27083]: audit 2024-04-24T17:29:54.814568+0000 mon.a (mon.0) 305 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T17:29:55.367 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:55 smithi032 bash[27083]: audit 2024-04-24T17:29:54.814568+0000 mon.a (mon.0) 305 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T17:29:55.367 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:55 smithi032 bash[27083]: audit 2024-04-24T17:29:54.816101+0000 mon.a (mon.0) 306 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-24T17:29:55.367 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:55 smithi032 bash[27083]: audit 2024-04-24T17:29:54.816101+0000 mon.a (mon.0) 306 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-24T17:29:55.367 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:55 smithi032 bash[27083]: audit 2024-04-24T17:29:54.818857+0000 mon.a (mon.0) 307 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T17:29:55.367 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:55 smithi032 bash[27083]: audit 2024-04-24T17:29:54.818857+0000 mon.a (mon.0) 307 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T17:29:55.367 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:55 smithi032 bash[27083]: audit 2024-04-24T17:29:54.819724+0000 mon.a (mon.0) 308 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:55.367 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:55 smithi032 bash[27083]: audit 2024-04-24T17:29:54.819724+0000 mon.a (mon.0) 308 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:55.367 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:55 smithi032 bash[21457]: audit 2024-04-24T17:29:54.805546+0000 mon.a (mon.0) 302 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:55.367 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:55 smithi032 bash[21457]: audit 2024-04-24T17:29:54.805546+0000 mon.a (mon.0) 302 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:55.367 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:55 smithi032 bash[21457]: audit 2024-04-24T17:29:54.806706+0000 mon.a (mon.0) 303 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T17:29:55.367 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:55 smithi032 bash[21457]: audit 2024-04-24T17:29:54.806706+0000 mon.a (mon.0) 303 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T17:29:55.367 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:55 smithi032 bash[21457]: audit 2024-04-24T17:29:54.812397+0000 mon.a (mon.0) 304 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:55.368 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:55 smithi032 bash[21457]: audit 2024-04-24T17:29:54.812397+0000 mon.a (mon.0) 304 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:55.368 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:55 smithi032 bash[21457]: audit 2024-04-24T17:29:54.814568+0000 mon.a (mon.0) 305 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T17:29:55.368 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:55 smithi032 bash[21457]: audit 2024-04-24T17:29:54.814568+0000 mon.a (mon.0) 305 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T17:29:55.368 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:55 smithi032 bash[21457]: audit 2024-04-24T17:29:54.816101+0000 mon.a (mon.0) 306 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-24T17:29:55.368 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:55 smithi032 bash[21457]: audit 2024-04-24T17:29:54.816101+0000 mon.a (mon.0) 306 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-24T17:29:55.368 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:55 smithi032 bash[21457]: audit 2024-04-24T17:29:54.818857+0000 mon.a (mon.0) 307 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T17:29:55.368 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:55 smithi032 bash[21457]: audit 2024-04-24T17:29:54.818857+0000 mon.a (mon.0) 307 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T17:29:55.368 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:55 smithi032 bash[21457]: audit 2024-04-24T17:29:54.819724+0000 mon.a (mon.0) 308 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:55.368 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:55 smithi032 bash[21457]: audit 2024-04-24T17:29:54.819724+0000 mon.a (mon.0) 308 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:55.368 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:55 smithi032 systemd[1]: Stopping Ceph mon.a for a7981b84-025f-11ef-bc93-c7b262605968... 2024-04-24T17:29:55.368 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:55 smithi032 bash[21457]: debug 2024-04-24T17:29:55.305+0000 7f7ecc132700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-04-24T17:29:55.368 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 24 17:29:55 smithi032 bash[21457]: debug 2024-04-24T17:29:55.305+0000 7f7ecc132700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-24T17:29:55.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:55 smithi082 bash[21793]: audit 2024-04-24T17:29:54.805546+0000 mon.a (mon.0) 302 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:55.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:55 smithi082 bash[21793]: audit 2024-04-24T17:29:54.805546+0000 mon.a (mon.0) 302 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:55.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:55 smithi082 bash[21793]: audit 2024-04-24T17:29:54.806706+0000 mon.a (mon.0) 303 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T17:29:55.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:55 smithi082 bash[21793]: audit 2024-04-24T17:29:54.806706+0000 mon.a (mon.0) 303 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T17:29:55.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:55 smithi082 bash[21793]: audit 2024-04-24T17:29:54.812397+0000 mon.a (mon.0) 304 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:55.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:55 smithi082 bash[21793]: audit 2024-04-24T17:29:54.812397+0000 mon.a (mon.0) 304 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' 2024-04-24T17:29:55.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:55 smithi082 bash[21793]: audit 2024-04-24T17:29:54.814568+0000 mon.a (mon.0) 305 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T17:29:55.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:55 smithi082 bash[21793]: audit 2024-04-24T17:29:54.814568+0000 mon.a (mon.0) 305 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T17:29:55.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:55 smithi082 bash[21793]: audit 2024-04-24T17:29:54.816101+0000 mon.a (mon.0) 306 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-24T17:29:55.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:55 smithi082 bash[21793]: audit 2024-04-24T17:29:54.816101+0000 mon.a (mon.0) 306 : audit [INF] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-24T17:29:55.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:55 smithi082 bash[21793]: audit 2024-04-24T17:29:54.818857+0000 mon.a (mon.0) 307 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T17:29:55.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:55 smithi082 bash[21793]: audit 2024-04-24T17:29:54.818857+0000 mon.a (mon.0) 307 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T17:29:55.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:55 smithi082 bash[21793]: audit 2024-04-24T17:29:54.819724+0000 mon.a (mon.0) 308 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:55.380 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:55 smithi082 bash[21793]: audit 2024-04-24T17:29:54.819724+0000 mon.a (mon.0) 308 : audit [DBG] from='mgr.14152 v1:172.21.15.32:0/1117817972' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T17:29:56.107 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a7981b84-025f-11ef-bc93-c7b262605968@mon.a.service' 2024-04-24T17:29:56.139 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T17:29:56.139 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-24T17:29:56.140 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-04-24T17:29:56.140 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-a7981b84-025f-11ef-bc93-c7b262605968@mon.c 2024-04-24T17:29:56.377 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:56 smithi032 systemd[1]: Stopping Ceph mon.c for a7981b84-025f-11ef-bc93-c7b262605968... 2024-04-24T17:29:56.378 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:56 smithi032 bash[27083]: debug 2024-04-24T17:29:56.309+0000 7f167c2fa700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-04-24T17:29:56.378 INFO:journalctl@ceph.mon.c.smithi032.stdout:Apr 24 17:29:56 smithi032 bash[27083]: debug 2024-04-24T17:29:56.309+0000 7f167c2fa700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-04-24T17:29:56.894 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:56 smithi082 systemd[1]: /etc/systemd/system/ceph-a7981b84-025f-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-24T17:29:56.895 INFO:journalctl@ceph.mgr.x.smithi082.stdout:Apr 24 17:29:56 smithi082 systemd[1]: /etc/systemd/system/ceph-a7981b84-025f-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-24T17:29:57.145 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a7981b84-025f-11ef-bc93-c7b262605968@mon.c.service' 2024-04-24T17:29:57.157 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:57 smithi082 systemd[1]: /etc/systemd/system/ceph-a7981b84-025f-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-24T17:29:57.157 INFO:journalctl@ceph.mgr.x.smithi082.stdout:Apr 24 17:29:57 smithi082 systemd[1]: /etc/systemd/system/ceph-a7981b84-025f-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-24T17:29:57.157 INFO:journalctl@ceph.mgr.x.smithi082.stdout:Apr 24 17:29:57 smithi082 systemd[1]: Started Ceph mgr.x for a7981b84-025f-11ef-bc93-c7b262605968. 2024-04-24T17:29:57.167 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T17:29:57.167 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-04-24T17:29:57.167 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-04-24T17:29:57.167 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-a7981b84-025f-11ef-bc93-c7b262605968@mon.b 2024-04-24T17:29:57.605 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:57 smithi082 systemd[1]: Stopping Ceph mon.b for a7981b84-025f-11ef-bc93-c7b262605968... 2024-04-24T17:29:57.879 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:57 smithi082 bash[21793]: debug 2024-04-24T17:29:57.601+0000 7f311d94b700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-04-24T17:29:57.880 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:57 smithi082 bash[21793]: debug 2024-04-24T17:29:57.601+0000 7f311d94b700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-04-24T17:29:58.593 INFO:journalctl@ceph.mon.b.smithi082.stdout:Apr 24 17:29:58 smithi082 bash[22911]: ceph-a7981b84-025f-11ef-bc93-c7b262605968-mon-b 2024-04-24T17:29:58.610 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a7981b84-025f-11ef-bc93-c7b262605968@mon.b.service' 2024-04-24T17:29:58.656 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T17:29:58.656 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-04-24T17:29:58.656 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-24T17:29:58.656 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-a7981b84-025f-11ef-bc93-c7b262605968@mgr.y 2024-04-24T17:29:58.999 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:29:58 smithi032 systemd[1]: Stopping Ceph mgr.y for a7981b84-025f-11ef-bc93-c7b262605968... 2024-04-24T17:29:59.000 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:29:58 smithi032 bash[21712]: debug 2024-04-24T17:29:58.801+0000 7fd402d2d700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-04-24T17:29:59.000 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 24 17:29:58 smithi032 bash[21712]: debug 2024-04-24T17:29:58.801+0000 7fd402d2d700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-24T17:29:59.539 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a7981b84-025f-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-24T17:29:59.566 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T17:29:59.566 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-24T17:29:59.567 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-24T17:29:59.567 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-a7981b84-025f-11ef-bc93-c7b262605968@mgr.x 2024-04-24T17:29:59.877 INFO:journalctl@ceph.mgr.x.smithi082.stdout:Apr 24 17:29:59 smithi082 systemd[1]: Stopping Ceph mgr.x for a7981b84-025f-11ef-bc93-c7b262605968... 2024-04-24T17:30:00.414 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a7981b84-025f-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-24T17:30:00.457 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T17:30:00.457 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-24T17:30:00.457 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a7981b84-025f-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-24T17:30:00.631 INFO:teuthology.orchestra.run.smithi032.stdout:Deleting cluster with fsid: a7981b84-025f-11ef-bc93-c7b262605968 2024-04-24T17:30:02.894 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a7981b84-025f-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-24T17:30:03.061 INFO:teuthology.orchestra.run.smithi082.stdout:Deleting cluster with fsid: a7981b84-025f-11ef-bc93-c7b262605968 2024-04-24T17:30:05.144 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-24T17:30:05.158 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-24T17:30:05.171 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-24T17:30:05.173 DEBUG:teuthology.misc:Transferring archived files from smithi032:/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/adking-2024-04-24_15:49:18-orch:cephadm-wip-adk-testing-2024-04-24-0937-distro-default-smithi/7671850/remote/smithi032/crash 2024-04-24T17:30:05.174 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/crash -- . 2024-04-24T17:30:05.213 INFO:teuthology.orchestra.run.smithi032.stderr:tar: /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-24T17:30:05.213 INFO:teuthology.orchestra.run.smithi032.stderr:tar: Error is not recoverable: exiting now 2024-04-24T17:30:05.215 DEBUG:teuthology.misc:Transferring archived files from smithi082:/var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/adking-2024-04-24_15:49:18-orch:cephadm-wip-adk-testing-2024-04-24-0937-distro-default-smithi/7671850/remote/smithi082/crash 2024-04-24T17:30:05.216 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar c -f - -C /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/crash -- . 2024-04-24T17:30:05.227 INFO:teuthology.orchestra.run.smithi082.stderr:tar: /var/lib/ceph/a7981b84-025f-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-24T17:30:05.227 INFO:teuthology.orchestra.run.smithi082.stderr:tar: Error is not recoverable: exiting now 2024-04-24T17:30:05.228 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-24T17:30:05.228 DEBUG:teuthology.orchestra.run.smithi032:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'reached quota' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | head -n 1 2024-04-24T17:30:05.271 INFO:teuthology.orchestra.run.smithi032.stdout:2024-04-24T17:28:34.629151+0000 mon.a (mon.0) 104 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED) 2024-04-24T17:30:05.272 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-04-24T17:30:05.272 DEBUG:teuthology.orchestra.run.smithi032:> sudo egrep '\[SEC\]' /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'reached quota' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | head -n 1 2024-04-24T17:30:05.326 DEBUG:teuthology.orchestra.run.smithi032:> sudo egrep '\[ERR\]' /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'reached quota' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | head -n 1 2024-04-24T17:30:05.381 DEBUG:teuthology.orchestra.run.smithi032:> sudo egrep '\[WRN\]' /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'reached quota' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | head -n 1 2024-04-24T17:30:05.436 INFO:teuthology.orchestra.run.smithi032.stdout:2024-04-24T17:28:34.629151+0000 mon.a (mon.0) 104 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED) 2024-04-24T17:30:05.437 INFO:tasks.cephadm:Compressing logs... 2024-04-24T17:30:05.438 DEBUG:teuthology.orchestra.run.smithi032:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-24T17:30:05.482 DEBUG:teuthology.orchestra.run.smithi082:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-24T17:30:05.491 INFO:teuthology.orchestra.run.smithi032.stderr:find: '/var/log/rbd-target-api': No such file or directory 2024-04-24T17:30:05.498 INFO:teuthology.orchestra.run.smithi082.stderr:find: '/var/log/rbd-target-api': No such file or directory 2024-04-24T17:30:05.498 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-24T17:30:05.498 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-24T17:30:05.499 INFO:teuthology.orchestra.run.smithi032.stderr:gzip/var/log/ceph/cephadm.log: -5 --verbose -- /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-24T17:30:05.499 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-24T17:30:05.500 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968/ceph.log 2024-04-24T17:30:05.500 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-24T17:30:05.500 INFO:teuthology.orchestra.run.smithi032.stderr: 83.6% -- replaced with /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-24T17:30:05.501 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-24T17:30:05.501 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968/ceph-mon.c.log: 92.5% -- replaced with /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-24T17:30:05.502 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968/ceph.audit.log: 90.6% -- replaced with /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-24T17:30:05.505 INFO:teuthology.orchestra.run.smithi032.stderr: 91.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-24T17:30:05.513 INFO:teuthology.orchestra.run.smithi032.stderr: 91.5% -- replaced with /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-24T17:30:05.514 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-24T17:30:05.514 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-24T17:30:05.515 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968/ceph.log 2024-04-24T17:30:05.515 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968/ceph.cephadm.log: 72.7% -- replaced with /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-24T17:30:05.515 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-24T17:30:05.516 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-24T17:30:05.516 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968/ceph.log: /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968/ceph.audit.log: 88.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-24T17:30:05.516 INFO:teuthology.orchestra.run.smithi082.stderr: 88.6% -- replaced with /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-24T17:30:05.516 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968/ceph-mon.b.log: 88.8% -- replaced with /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-24T17:30:05.525 INFO:teuthology.orchestra.run.smithi082.stderr: 91.4% -- replaced with /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-24T17:30:05.527 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-04-24T17:30:05.527 INFO:teuthology.orchestra.run.smithi082.stderr:real 0m0.039s 2024-04-24T17:30:05.527 INFO:teuthology.orchestra.run.smithi082.stderr:user 0m0.023s 2024-04-24T17:30:05.527 INFO:teuthology.orchestra.run.smithi082.stderr:sys 0m0.016s 2024-04-24T17:30:05.532 INFO:teuthology.orchestra.run.smithi032.stderr: 89.7% -- replaced with /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-24T17:30:05.573 INFO:teuthology.orchestra.run.smithi032.stderr: 90.8% -- replaced with /var/log/ceph/a7981b84-025f-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-24T17:30:05.574 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-04-24T17:30:05.574 INFO:teuthology.orchestra.run.smithi032.stderr:real 0m0.090s 2024-04-24T17:30:05.575 INFO:teuthology.orchestra.run.smithi032.stderr:user 0m0.130s 2024-04-24T17:30:05.575 INFO:teuthology.orchestra.run.smithi032.stderr:sys 0m0.018s 2024-04-24T17:30:05.575 INFO:tasks.cephadm:Archiving logs... 2024-04-24T17:30:05.575 DEBUG:teuthology.misc:Transferring archived files from smithi032:/var/log/ceph to /home/teuthworker/archive/adking-2024-04-24_15:49:18-orch:cephadm-wip-adk-testing-2024-04-24-0937-distro-default-smithi/7671850/remote/smithi032/log 2024-04-24T17:30:05.576 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-24T17:30:05.647 DEBUG:teuthology.misc:Transferring archived files from smithi082:/var/log/ceph to /home/teuthworker/archive/adking-2024-04-24_15:49:18-orch:cephadm-wip-adk-testing-2024-04-24-0937-distro-default-smithi/7671850/remote/smithi082/log 2024-04-24T17:30:05.648 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-24T17:30:05.667 INFO:tasks.cephadm:Removing cluster... 2024-04-24T17:30:05.668 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a7981b84-025f-11ef-bc93-c7b262605968 --force 2024-04-24T17:30:05.837 INFO:teuthology.orchestra.run.smithi032.stdout:Deleting cluster with fsid: a7981b84-025f-11ef-bc93-c7b262605968 2024-04-24T17:30:07.161 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a7981b84-025f-11ef-bc93-c7b262605968 --force 2024-04-24T17:30:07.331 INFO:teuthology.orchestra.run.smithi082.stdout:Deleting cluster with fsid: a7981b84-025f-11ef-bc93-c7b262605968 2024-04-24T17:30:08.672 INFO:tasks.cephadm:Removing cephadm ... 2024-04-24T17:30:08.672 DEBUG:teuthology.orchestra.run.smithi032:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-24T17:30:08.678 DEBUG:teuthology.orchestra.run.smithi082:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-24T17:30:08.684 INFO:tasks.cephadm:Teardown complete 2024-04-24T17:30:08.684 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_5904d29475f5be602879d9fb26280e89b808d5cc/qa/tasks/cephadm.py", line 2220, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_5904d29475f5be602879d9fb26280e89b808d5cc/qa/tasks/cephadm.py", line 1849, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_5904d29475f5be602879d9fb26280e89b808d5cc/qa/tasks/cephadm.py", line 1091, in ceph_osds remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi032 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a7981b84-025f-11ef-bc93-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4' 2024-04-24T17:30:08.896 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=1a65451903364a3887f2f4bd20bd2326 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_5904d29475f5be602879d9fb26280e89b808d5cc/qa/tasks/cephadm.py", line 2220, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_5904d29475f5be602879d9fb26280e89b808d5cc/qa/tasks/cephadm.py", line 1849, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_5904d29475f5be602879d9fb26280e89b808d5cc/qa/tasks/cephadm.py", line 1091, in ceph_osds remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi032 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5904d29475f5be602879d9fb26280e89b808d5cc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a7981b84-025f-11ef-bc93-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4' 2024-04-24T17:30:08.901 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-24T17:30:08.910 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-24T17:30:08.944 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-24T17:30:08.944 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-24T17:30:08.959 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-24T17:30:09.023 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-24T17:30:09.032 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-24T17:30:09.032 DEBUG:teuthology.orchestra.run.smithi032:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T17:30:09.034 DEBUG:teuthology.orchestra.run.smithi082:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T17:30:09.046 INFO:teuthology.orchestra.run.smithi032.stdout: remote refid st t when poll reach delay offset jitter 2024-04-24T17:30:09.046 INFO:teuthology.orchestra.run.smithi032.stdout:============================================================================== 2024-04-24T17:30:09.046 INFO:teuthology.orchestra.run.smithi032.stdout:+hv01.front.sepi 44.4.53.2 2 u 62 64 77 0.134 -2.137 1.115 2024-04-24T17:30:09.046 INFO:teuthology.orchestra.run.smithi032.stdout:*hv02.front.sepi 63.231.80.2 3 u 61 64 77 0.096 -0.398 1.075 2024-04-24T17:30:09.046 INFO:teuthology.orchestra.run.smithi032.stdout:+hv03.front.sepi 74.6.168.72 3 u 57 64 77 0.149 -0.115 1.087 2024-04-24T17:30:09.047 INFO:teuthology.orchestra.run.smithi032.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-24T17:30:09.048 INFO:teuthology.orchestra.run.smithi082.stdout: remote refid st t when poll reach delay offset jitter 2024-04-24T17:30:09.048 INFO:teuthology.orchestra.run.smithi082.stdout:============================================================================== 2024-04-24T17:30:09.048 INFO:teuthology.orchestra.run.smithi082.stdout:+hv01.front.sepi 44.4.53.2 2 u 63 64 77 0.077 -2.275 0.980 2024-04-24T17:30:09.049 INFO:teuthology.orchestra.run.smithi082.stdout:*hv02.front.sepi 63.231.80.2 3 u 63 64 77 0.142 -0.530 0.950 2024-04-24T17:30:09.049 INFO:teuthology.orchestra.run.smithi082.stdout:+hv03.front.sepi 74.6.168.72 3 u 62 64 77 0.100 -0.348 0.948 2024-04-24T17:30:09.049 INFO:teuthology.orchestra.run.smithi082.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-24T17:30:09.049 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-24T17:30:09.060 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-24T17:30:09.060 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-24T17:30:09.095 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-24T17:30:09.128 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-24T17:30:09.161 INFO:teuthology.task.internal:Duration was 826.442875 seconds 2024-04-24T17:30:09.161 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-24T17:30:09.195 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-24T17:30:09.195 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-24T17:30:09.198 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-24T17:30:09.236 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-24T17:30:09.237 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi032.front.sepia.ceph.com 2024-04-24T17:30:09.237 DEBUG:teuthology.orchestra.run.smithi032:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-24T17:30:09.290 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi082.front.sepia.ceph.com 2024-04-24T17:30:09.291 DEBUG:teuthology.orchestra.run.smithi082:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-24T17:30:09.301 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-24T17:30:09.301 DEBUG:teuthology.orchestra.run.smithi032:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-24T17:30:09.334 DEBUG:teuthology.orchestra.run.smithi082:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-24T17:30:09.372 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-24T17:30:09.372 DEBUG:teuthology.orchestra.run.smithi032:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-24T17:30:09.411 DEBUG:teuthology.orchestra.run.smithi082:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-24T17:30:09.493 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-24T17:30:09.504 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-24T17:30:09.504 DEBUG:teuthology.orchestra.run.smithi032:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-24T17:30:09.518 DEBUG:teuthology.orchestra.run.smithi082:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-24T17:30:09.531 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-24T17:30:09.540 DEBUG:teuthology.orchestra.run.smithi032:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-24T17:30:09.562 DEBUG:teuthology.orchestra.run.smithi082:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-24T17:30:09.572 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern = core 2024-04-24T17:30:09.585 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern = core 2024-04-24T17:30:09.615 DEBUG:teuthology.orchestra.run.smithi032:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-24T17:30:09.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T17:30:09.661 DEBUG:teuthology.orchestra.run.smithi082:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-24T17:30:09.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T17:30:09.666 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-24T17:30:09.675 INFO:teuthology.task.internal:Transferring archived files... 2024-04-24T17:30:09.676 DEBUG:teuthology.misc:Transferring archived files from smithi032:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/adking-2024-04-24_15:49:18-orch:cephadm-wip-adk-testing-2024-04-24-0937-distro-default-smithi/7671850/remote/smithi032 2024-04-24T17:30:09.676 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-24T17:30:09.731 DEBUG:teuthology.misc:Transferring archived files from smithi082:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/adking-2024-04-24_15:49:18-orch:cephadm-wip-adk-testing-2024-04-24-0937-distro-default-smithi/7671850/remote/smithi082 2024-04-24T17:30:09.731 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-24T17:30:09.753 INFO:teuthology.task.internal:Removing archive directory... 2024-04-24T17:30:09.753 DEBUG:teuthology.orchestra.run.smithi032:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-24T17:30:09.771 DEBUG:teuthology.orchestra.run.smithi082:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-24T17:30:09.796 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-24T17:30:09.806 INFO:teuthology.task.internal:Not uploading archives. 2024-04-24T17:30:09.806 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-24T17:30:09.828 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-24T17:30:09.829 DEBUG:teuthology.orchestra.run.smithi032:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-24T17:30:09.831 DEBUG:teuthology.orchestra.run.smithi082:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-24T17:30:09.834 INFO:teuthology.orchestra.run.smithi032.stdout: 658618 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 24 17:30 /home/ubuntu/cephtest 2024-04-24T17:30:09.843 INFO:teuthology.orchestra.run.smithi082.stdout: 658467 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 24 17:30 /home/ubuntu/cephtest 2024-04-24T17:30:09.845 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-24T17:30:09.862 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-24T17:30:09.928 INFO:teuthology.nuke:Checking targets against current locks 2024-04-24T17:30:09.955 DEBUG:teuthology.nuke:shortname: smithi032 2024-04-24T17:30:09.955 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-24T17:30:09.977 DEBUG:teuthology.nuke:shortname: smithi082 2024-04-24T17:30:09.977 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-24T17:30:09.981 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi032.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-04-24_15:49:18-orch:cephadm-wip-adk-testing-2024-04-24-0937-distro-default-smithi/7671850', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 17:08:05.312896', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJhPwZXiDgl9ZAcEizV9fY7iWJ/ltclp1IPZAitkGKbEdcOM0LSRMmFZruXmmTE9TxQifb3wjzmfKGPW1fHn2fk='} 2024-04-24T17:30:10.002 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi082.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-04-24_15:49:18-orch:cephadm-wip-adk-testing-2024-04-24-0937-distro-default-smithi/7671850', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 17:08:05.314199', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKpPIX8xrVbszAlxPyVcXR2kbxZ9olpvCuOj9FIZgFXyNL+DZ3Cy2N8pApUd0OYdmL4z6aEcIt5v9lxIXM6PipU='} 2024-04-24T17:30:10.033 INFO:teuthology.orchestra.console.smithi032:Power off 2024-04-24T17:30:10.033 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-24T17:30:10.055 DEBUG:teuthology.orchestra.console.smithi032:power off output: Chassis Power Control: Down/Off 2024-04-24T17:30:10.056 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T17:30:10.070 INFO:teuthology.orchestra.console.smithi082:Power off 2024-04-24T17:30:10.071 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-24T17:30:10.079 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is on 2024-04-24T17:30:10.092 DEBUG:teuthology.orchestra.console.smithi082:power off output: Chassis Power Control: Down/Off 2024-04-24T17:30:10.092 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T17:30:10.113 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is on 2024-04-24T17:30:14.081 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T17:30:14.114 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T17:30:14.192 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is on 2024-04-24T17:30:14.224 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is on 2024-04-24T17:30:18.194 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T17:30:18.224 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T17:30:18.305 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is off 2024-04-24T17:30:18.334 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is off 2024-04-24T17:30:18.406 INFO:teuthology.orchestra.console.smithi032:Power off completed 2024-04-24T17:30:18.435 INFO:teuthology.orchestra.console.smithi082:Power off completed 2024-04-24T17:30:18.540 INFO:teuthology.run:Summary data: description: orch:cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} duration: 826.4428753852844 failure_reason: '"2024-04-24T17:28:34.629151+0000 mon.a (mon.0) 104 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log' flavor: default owner: scheduled_adking@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=1a65451903364a3887f2f4bd20bd2326 status: fail success: false 2024-04-24T17:30:18.540 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-24T17:30:18.619 INFO:teuthology.run:FAIL